ORA-12844: cluster reconfiguration in progress
Hi All,
I am getting ORA-12844: cluster reconfiguration in progress error while executing a mapping with UPDATE/INSERT option. For the first time, the mapping was executed with out any problem and loaded about 16 million records.. For the subsequent executions it is giving the above problem.
What would be the problem? please respond, if anyone came across the similar situation.
Thanks,
PR
1. Are you running against Oracle RAC? The message suggest that but we've seen this issue once before when it wasn't related to RAC (no good suggestions from that occurrence).
2. Have you ever seen this message outside the context of what you do with OWB? OWB should not do anything special that would cause this issue.
Nikolai Rochnik
Similar Messages
-
ORA-15137: cluster in rolling patch
Hello everyone,
something went wrong in patching grid 12.1.0.2 with patch: Bug 19954978 - GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.2 (JAN2015) (linux 64)
And when I try to create a diskgroup I get:
ORA-15018: Diskgroup can not be created
ORA-15137: cluster in rolling patch
Although opatch lsinventory shows succeeded patching
Can someone help ?
How do I tell the clusterware to quit rolling patch ?
Please help.
PaoloThak you, here is what I get:
[root@ap2-oraclec01 bin]# ./crsctl query crs softwarepatch ap2-oraclec00
Oracle Clusterware patch level on node ap2-oraclec00 is [0].
[root@ap2-oraclec01 bin]# ./crsctl query crs softwarepatch ap2-oraclec01
Oracle Clusterware patch level on node ap2-oraclec01 is [0].
[root@ap2-oraclec01 bin]# ./crsctl stop rollingpatch
CRS-1170: Rejecting rolling patch mode change because the patch has not been applied yet. The software patch level [0] on this node is not the same as the expected patch level [1964077727].
CRS-4000: Command Stop failed, or completed with errors.
[root@ap2-oraclec01 bin]# -
Hi all,
We have moved one Sun Cluster 3.0 (Solaris 8, and HA-Oracle) to a new datacenter so all the storage have changed from one EMC symmetrics to a new one.
We decided to start all the services manually (boot -x, importt disk groups, etc) to study the new situation and make a workaround to reconfigure the cluster.
Have any idea of the problems we can encounter? As we can se, we have to discover the new disks (scdidadm -r) and change the quorum device
ByeWe don't see both symmetrics now.
When the migration was done both DMX were syncronized. We stopped one node and moved it to the new datacenter. After that, the application was stopped and the disks were 'splitted'. After that the application was started importing de disk groups.
The problem is the following: we've got a cluster stopped but al the DID configuration has changed (includind quorum disk) because the new disks are availabre through a new path (corresponding to the new fiber path)
For example:
# scdidadm -l
3 peai01:/dev/rdsk/c5t500604843E0C1412d11 /dev/did/rdsk/d3
That disk doesn't exist and now it's named "c5t500604843E0C2713d11s2". That happens with all the disk.
To make 'scdidadm -r' we need the cluster up and running but that's not the situation -
Offline Backup failed : ORA-00704, ORA-00604, ORA-01089 Errors
Hi Guys
I am facing issues with Offline backup in Production system.
I had Scheduled the offline backup for sunday evening, but monday morning i found prodution box, was down and i killed the netbackup Process.
checked Oracle was up.
Just started SAP and system start running fine.
Then found that backup (offline backup) was never started.
Found following errors in alert_<sid>.log and in *.trc files:
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01089: immediate shutdown in progress - no operations are permitted
Oracle 9.2.0.7
Windows 2000
Can you guide me on this, as this has happened second time ?
RegardsYour database was trying to shut down, but in was hanging for some reason.
not much can be done, you are using Oracle 9207, try with 9208
you ware on an unsupported Windows 2000 version. -
ORA-02068: following severe error from - How to handle the overgiven Except
Hello again ^^,
hope its okay to write the follwnig question into a new Thread...
I sometimes get errors from remote dbs.
ORA-02068: following severe error from DBLINK
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-02068: following severe error from DBLINK
ORA-03113: end-of-file on communication channelThese are in one sqlerm. Is it possible to handle the second errors without string operations?
Thanks and regards
Edited by: user10283133 on 28.10.2008 08:33
Try to force the headlineThe remote db is accessible from 6 am to 6 pm. When querring longer than given time span I get the Error ORA-02068. In the sqlerm for this Error is a second Errormessage from an second different Error concatenated.
Now I will tell what I think is happening:
I think the remote db say: >>I will go sleep now<< an throws Error A (ORA-01089 (or ORA-03113, or what else)). This Error A goes to the local db. The local db get the Error A and throw a new Error (Error B = ORA-02068). This Error B is independet from Error A. And I see a String which is "Error B + Error A"
I recieve also three possible Errormessages:
ORA-02068: following severe error from DBLINK
ORA-03113: end-of-file on communication channel
ORA-02068: following severe error from DBLINK_ODS_STBY
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-02068: following severe error from DBLINK_ODS_STBY
ORA-whatelse: what elseI can handle the ORA-02068. But my questions is, can I handle the second Error (thus Error A) without string operations?
The Relase is 10g2. Please tell me which information you still need ...
Thank you very much.
regrads -
Shutdown immediate hangs ORA-12012, ORA-12008, ORA-06512
We have a problem while immediate shutting down the database:
alert.log:
Errors in file /data_archiv/oracle/admin/EGB/bdump/egb_j001_7708.trc:
ORA-12012: error on auto execute of job 3
ORA-12008: error in materialized view refresh path
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 1883
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 2089
ORA-06512: at "SYS.DBMS_IREFRESH", line 683
ORA-06512: at "SYS.DBMS_REFRESH", line 195
ORA-06512: at line 1
Mon Jun 11 16:20:29 2007
Job queue slave processes stopped
Mon Jun 11 16:26:36 2007
Active call for process 7695 user 'oracle' program 'oracle@lxegbp2qs (J000)'
SHUTDOWN: waiting for active calls to complete.
Trace:
/data_archiv/oracle/admin/EGB/bdump/egb_j001_7708.trc
Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /data_archiv/oracle/product/10g
System name: Linux
Node name: lxegbp2qs
Release: 2.6.5-7.244-smp
Version: #1 SMP Mon Dec 12 18:32:25 UTC 2005
Machine: i686
Instance name: EGB
Redo thread mounted by this instance: 1
Oracle process number: 19
Unix process pid: 7708, image: oracle@lxegbp2qs (J001)
*** SERVICE NAME:(SYS$USERS) 2007-06-11 16:20:28.759
*** SESSION ID:(273.1) 2007-06-11 16:20:28.759
*** 2007-06-11 16:20:28.759
ORA-12012: error on auto execute of job 3
ORA-12008: error in materialized view refresh path
ORA-01089: immediate shutdown in progress - no operations are permitted
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 1883
ORA-06512: at "SYS.DBMS_SNAPSHOT", line 2089
ORA-06512: at "SYS.DBMS_IREFRESH", line 683
ORA-06512: at "SYS.DBMS_REFRESH", line 195
ORA-06512: at line 1
The job with id 3 should not run during shut down (needs only seconds):
select substr(what, 1, 70), TO_CHAR(last_date, 'DD.MM.YYYY HH24:MI:SS') as last_date, TO_CHAR(next_date, 'DD.MM.YYYY SQL> HH24:MI:SS') as next_date from dba_jobs where job=3;
SUBSTR(WHAT,1,70)
LAST_DATE NEXT_DATE
dbms_refresh.refresh('"TEST"."MV_TEST"');
11.06.2007 06:00:06 12.06.2007 06:00:00
Any ideas where to look for?Active call for process 7695 user 'oracle' programKill active call process mentioned in alert.log.
-
Logs for Cluster Synchronization Service daemon
ORA-29746: Cluster Synchronization Service is being shut down.
Cause: The administrator has shut down the Cluster Synchronization Service daemon. This error message is intended to be informative to users on the status of the service.
Action: Check the log file of the Cluster Synchronization Service daemon to verify the state of the service.
Can someone tell me the location of these logs?
Thnaks$CRS_HOME/log/<node>/cssd/
-
Issues creating VSA cluster "Unknown value" error
Hi all,
I'm in the process of evaluating vSphere with VSA and running into an issue during creation of the VSA cluster. On the final step of creating the VSA cluster, I get an error message "Cannot create VSA cluster: Unknown value."
I'm setting up a two node cluster using a couple of HP ML310e Gen8's, using local storage and some extra networking via a quad-port NIC. Memory, CPU clock speed, and number of NICs is compliant with VSA requirements.
I've already double checked things like default gateway on the Management Network, naming of the various port groups to comply with what vSphere/VSA expects, specific NIC teaming requirements, and increased maximum heap size to 256 & set evc.config.baseline=highest in dev.properties as I've got a couple of VMs running on the two ESX servers (just Domain Controller and vCenter VMs).
Per the screenshots below, I've setup vSwitch0 and vSwitch1 as per VMware recommendations.
On vSwitch0:
Management Network and VSA-Front End portgroups are using native VLAN (192.168.1.0/24).
VM Network is using VLAN 110 (192.168.110.0/24).
vSwitch1 is fairly self-explanatory.
Only unique thing is that I've created 'vCenter VM VLAN' on vSwitch0 as a Virtual Machine portgroup so that I could assign this to the vCenter VM NIC as I don't have routing/ACLs in place between VM Network (VLAN 110) and Management Network (native VLAN).
Also, on ESX node # 2 I've got an issue with the physical NIC's associated with vmnic1 and vmnic3 (they seems to hang my Cisco switch each time I plug them in) so I've used the physical NIC's associated with vmnic4 and vmnic5 instead.
I've already searched the various VSA related threads but to no avail.
Hoping someone may be able to shed some light on this one.
Last few lines of the VSAManager.log:
2013-04-28 22:08:40,229 45 [DefaultJobRunner] [pool-140-thread-1] INFO - Starting job. JobName:ClusterCreation-datacenter-21
2013-04-28 22:08:40,229 58 [DefaultJobRunner] [pool-140-thread-1] INFO - Starting Task group. TaskGroupName:ClusterCreation-PreHostReboot
2013-04-28 22:08:40,229 49 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task begin. TaskID:TaskInitCreateCluster
2013-04-28 22:08:40,229 69 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting execution of task. TaskID:TaskInitCreateCluster
2013-04-28 22:08:40,416 157 [SvaSolutionLicenseManager] [pool-140-thread-1] INFO - AcquireLicense - Updating license count to : 1
2013-04-28 22:08:40,494 353 [Utils] [pool-140-thread-1] INFO - Persisting host: host-74
2013-04-28 22:08:40,494 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-Back End @ hostmoref: host-74
2013-04-28 22:08:40,510 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-Front End @ hostmoref: host-74
2013-04-28 22:08:40,526 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-NFS @ hostmoref: host-74
2013-04-28 22:08:40,541 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-VMotion @ hostmoref: host-74
2013-04-28 22:08:40,557 353 [Utils] [pool-140-thread-1] INFO - Persisting host: host-66
2013-04-28 22:08:40,557 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-Back End @ hostmoref: host-66
2013-04-28 22:08:40,572 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-Front End @ hostmoref: host-66
2013-04-28 22:08:40,572 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-NFS @ hostmoref: host-66
2013-04-28 22:08:40,588 361 [Utils] [pool-140-thread-1] INFO - Persisting svaNetwork: VSA-VMotion @ hostmoref: host-66
2013-04-28 22:08:40,604 71 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task execution finished with result:true. TaskID:TaskInitCreateCluster
2013-04-28 22:08:40,604 73 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task end. TaskID:TaskInitCreateCluster
2013-04-28 22:08:40,604 49 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task begin. TaskID:LogParams
2013-04-28 22:08:40,604 69 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting execution of task. TaskID:LogParams
2013-04-28 22:08:40,604 101 [TaskLogClusterCreationParams] [pool-140-thread-1] INFO - Dump VSA Cluster Configuration
VSA Management Network Settings
IP Address Netmask Gateway/Interface
Cluster : 192.168.1.30 255.255.255.0 192.168.1.254
VSAManager Service: 192.168.1.31 255.255.255.0 Local Area Connection
VSA VM Network Settings : host-74
IP Address Netmask Gateway Vlan ID
Management: 192.168.1.50 255.255.255.0 192.168.1.254 0
NFS : 192.168.1.51 255.255.255.0 192.168.1.254 0
Backend : 192.168.104.2 255.255.255.0 - 104
Feature : 192.168.105.2 255.255.255.0 192.168.1.254 0
VSA VM Network Settings : host-66
IP Address Netmask Gateway Vlan ID
Management: 192.168.1.40 255.255.255.0 192.168.1.254 0
NFS : 192.168.1.41 255.255.255.0 192.168.1.254 0
Backend : 192.168.104.1 255.255.255.0 - 104
Feature : 192.168.105.1 255.255.255.0 192.168.1.254 0
OVF Location: file:///C:/Program Files/VMware/Infrastructure/tomcat/webapps/VSAManager/ovf/sva_OVF10.ovf
Eager Zero : false
2013-04-28 22:08:40,604 71 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task execution finished with result:true. TaskID:LogParams
2013-04-28 22:08:40,604 73 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task end. TaskID:LogParams
2013-04-28 22:08:40,604 49 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task begin. TaskID:ClusterExists
2013-04-28 22:08:40,604 69 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting execution of task. TaskID:ClusterExists
2013-04-28 22:08:40,635 427 [SessionService] [pool-140-thread-1] INFO - Cluster create in progress for DC: datacenter-21
2013-04-28 22:08:40,635 71 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task execution finished with result:true. TaskID:ClusterExists
2013-04-28 22:08:40,635 73 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task end. TaskID:ClusterExists
2013-04-28 22:08:40,635 49 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task begin. TaskID:AuditHostGroup
2013-04-28 22:08:40,635 69 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting execution of task. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 235 [Utils] [pool-140-thread-1] INFO - Retrieve dev.property: evc.config=true
2013-04-28 22:08:43,614 235 [Utils] [pool-140-thread-1] INFO - Retrieve dev.property: evc.config.baseline=highest
2013-04-28 22:08:43,614 89 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task exception. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 80 [DefaultJobRunner] [pool-140-thread-1] ERROR - Exception caught. Starting rollback for job if rollback support . JobName:ClusterCreation-datacenter-21
SvaManagerException: Unknown value.
at com.vmware.sva.manager.audit.VCHostAudit$AMD_EVC_MODE.getByValue(VCHostAudit.java:202)
at com.vmware.sva.manager.audit.VCHostAudit.auditEVCModeForHostMORs(VCHostAudit.java:1497)
at com.vmware.sva.manager.audit.VCHostAudit.auditEVCMode(VCHostAudit.java:1442)
at com.vmware.sva.manager.audit.VCHostAudit.auditHostGroup(VCHostAudit.java:1419)
at com.vmware.sva.manager.audit.AuditManager.auditHostGroup(AuditManager.java:152)
at com.vmware.sva.manager.jobservices.tasks.TaskAuditHostGroup.execute(TaskAuditHostGroup.java:48)
at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.handleTaskStart(DefaultTaskRunner.java:70)
at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.call(DefaultTaskRunner.java:33)
at com.vmware.sva.manager.jobservices.impl.SyncTaskGroup.execute(SyncTaskGroup.java:46)
at com.vmware.sva.manager.jobservices.impl.DefaultJobRunner.run(DefaultJobRunner.java:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-04-28 22:08:43,614 109 [DefaultJobRunner] [pool-140-thread-1] INFO - Rollback started for job. JobName:ClusterCreation-datacenter-21
2013-04-28 22:08:43,614 116 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 136 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting rollback of task. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 138 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task rollback finished with result:true. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 140 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback end. TaskID:AuditHostGroup
2013-04-28 22:08:43,614 116 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback. TaskID:ClusterExists
2013-04-28 22:08:43,614 136 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting rollback of task. TaskID:ClusterExists
2013-04-28 22:08:43,614 138 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task rollback finished with result:true. TaskID:ClusterExists
2013-04-28 22:08:43,614 140 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback end. TaskID:ClusterExists
2013-04-28 22:08:43,614 116 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback. TaskID:LogParams
2013-04-28 22:08:43,614 136 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting rollback of task. TaskID:LogParams
2013-04-28 22:08:43,614 138 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task rollback finished with result:true. TaskID:LogParams
2013-04-28 22:08:43,614 140 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback end. TaskID:LogParams
2013-04-28 22:08:43,614 116 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback. TaskID:TaskInitCreateCluster
2013-04-28 22:08:43,614 136 [DefaultTaskRunner] [pool-140-thread-1] INFO - Starting rollback of task. TaskID:TaskInitCreateCluster
2013-04-28 22:08:43,630 196 [MappingService] [pool-140-thread-1] INFO - Removing context for datacenter-21
2013-04-28 22:08:43,770 171 [SvaSolutionLicenseManager] [pool-140-thread-1] INFO - ReleaseLicense - Updating license count to : 0
2013-04-28 22:08:43,817 83 [TaskInitCreateCluster] [pool-140-thread-1] INFO - Released license for datacenter: datacenter-21
2013-04-28 22:08:43,817 138 [DefaultTaskRunner] [pool-140-thread-1] INFO - Task rollback finished with result:true. TaskID:TaskInitCreateCluster
2013-04-28 22:08:43,817 140 [DefaultTaskRunner] [pool-140-thread-1] INFO - Notifying listeners of task rollback end. TaskID:TaskInitCreateCluster
2013-04-28 22:08:43,817 122 [DefaultJobRunner] [pool-140-thread-1] INFO - TaskGroup rolled back successfully.TaskGroupName:ClusterCreation-PreHostReboot
2013-04-28 22:08:43,817 84 [DefaultJobRunner] [pool-140-thread-1] ERROR - Rollback finished successfully for job. JobName:ClusterCreation-datacenter-21
2013-04-28 22:08:43,817 92 [DefaultJobRunner] [pool-140-thread-1] INFO - Notifying listeners of job exception for job: ClusterCreation-datacenter-21
2013-04-28 22:08:45,549 196 [UserSession] [tomcat-http--20] INFO - **** extenMoref **** Datacenter:datacenter-21Hi,
I'm having the same problem :/
Did you find solution yet?
Kindest Regards -
Fresh cluster inst., servers reboot, cannot restart clusters or ASM
Hello all,
I just installed 11Gr2 cluster over 5 nodes. I used ASM in the installer, to hold the voting disk, etc for it.
I installed the RDBMS binairies successfully across all nodes. NO INSTANCES YET.
A few days went by....
I was getting ready to do post installing patches...and found things looking strange. I found the (working on node1), the clustering systems was not running.
I looked, and the servers (all 5 of them) for some reason had rebooted since install.
I tried starting the cluster:
crsctl start cluster -all.
Took while to return, and then errored with a timeout msg.
I checked to see if it was up:
./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4533: Event Manager is online
It then dawned on me...maybe ASM wasn't up either?
Nope...not running.
I tried to start it locally on node 1
I set the SID, and tried using sqlplus
I got:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/init+ASM1.ora'
I looked...nothing in that directory at all but a simple init.ora file.
I tried shutting down the cluster with
crsctl stop cluster -all
I got a ton of messages for each node like:
CRS-4548: Unable to connect to CRSD
CRS-2678: 'ora.crsd' on 'node1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
I'm trying to get through to Oracle support, but they're running slow.
Any ideas here?
I used OUI to create ASM for the cluster...why would it not put an init file there to point to the spfile in ASM?
I'm guessing this is the reason the nodes couldn't talk or sync. Trouble is...how do I start ASM without an init file? I seem to recall there might be a way to create a file to point to the ASM for the spfile, but I'm new to this too...and not sure where to point or the syntax to use.
Will have starting cluster up with no ASM have done any damage...if so, how to fix it?
As you can tell, learning about clusters/RAC and ASM....and I'm not finding good reference materials on troubleshooting. Heck, the install docs are bad enough....
Thank you in advance for any advice or links...
cayenne
ps. this is on RHEL5
Edited by: cayenne on Aug 10, 2010 12:33 PMssolbach wrote:
Hi,
Try doing crsctl stop crs -f 2 times. Sometimes this is needed.
crsctl check crs should report you that nothing is running anymore.
If it is... then only way I can think of is disable the automatic startup of crs
crsctl disable crs
and restart the node... This will definitely bring everything down (and does not start it up after restarting).
You can then start the crs stack with crsctl start crs.
Furthermore don't get confused with oracleasm.
Oracleasm is for asmlib. Which may be used in preparating the storage for ASM but has nothing to do later with if ASM is running or not.
To check the status of ASM first see if clusterstack was started successfully (you need to wait a little till it is started):
crsctl check crs.
Everything should be online... if it is, you can do a crsctl stat res -t which will show you all ressources including if ASM is running.
If there has been a problem starting up the stack (crsctl check crs) then we have to find out why.
Check $CRS_HOME/log/<hostname>/alert*.log for error messages.
If something indicates a problem with ASM do a:
adrci
show alert
and choose the ASM alert.log.
Search for error messages.
GL.
SebastianThank you, I'd not known of adrci before!! Once I set the ORACLE_HOME, it came right up!
Ok, it does look like ASM problem...and clustering can't find the voting disc.
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_25186.trc:
ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:sendmsg failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgxpsnd1
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_25186.trc:
ORA-27508: IPC error sending a message
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_o000_22504.trc (incident=4801):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 192.168.100.1 not found. Check output from ifconfig command
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_4801/+ASM1_o000_22504_i4801.trc
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_gmon_25212.trc:
ORA-29746: Cluster Synchronization Service is being shut down.
ORA-29702: error occurred in Cluster Group Service operation
GMON (ospid: 25212): terminating the instance due to error 29746
opidrv aborting process O000 ospid (22504) as a result of ORA-603
Ok, I've been going through the alert log, and the .trc files it indicates. I started by checking the private address 192.168.100.1...it seems to be up with ifconfig..and pingable.
I'm looking through errors...the one : ORA-27300 returning a value 22 got a hit on Oracle Support...but wasn't the same errors as I got.
Whew...still plowing through all the files and logs...not seeing anything out there so far that matches my problem...dunno what could have caused this to just all go BANG. I mean...no one but me using the machines, no databases on them yet...all that was installed was clustering with its voting disk and (ocr?) on a single ASM disk group....and RDBMS binaries installed across all 5 nodes.
Please let me know if you see anything that stands out here...I'm still searching myself...
Again, thanks for the adrcpi hint!!
cayenne -
Channel stopped by administrative task - ADAPTER.JAVA_EXCEPTION
Hi Guys,
Adapter engine at RWB has the following error message.
==== Events on Server 44769350 ====
Adapter Engine, message: Fatal DeleteJob (Default Delete Job) failed.
(Details: Code: MS.JOB.DELETE, Location: com.sap.aii.af.ra.ms.impl.core.archiving.executor.DeleteExecutor, Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Failed to delete unassigned audit log entry keys. Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Could not get unassigned audit log entry keys, due to: ORA-01089: immediate shutdown in progress - no operations are permitted
, Time: 08.03.2009 10:00:15)
Adapter Engine, message: Fatal DeleteJob (Default Delete Job) failed.
(Details: Code: MS.JOB.DELETE, Location: com.sap.aii.af.ra.ms.impl.core.archiving.executor.DeleteExecutor, Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Failed to delete unassigned audit log entry keys. Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Could not get unassigned audit log entry keys, due to: ORA-01089: immediate shutdown in progress - no operations are permitted
, Time: 15.02.2009 10:00:11)
Adapter Engine, message: Fatal DeleteJob (Default Delete Job) failed.
(Details: Code: MS.JOB.DELETE, Location: com.sap.aii.af.ra.ms.impl.core.archiving.executor.DeleteExecutor, Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Failed to delete unassigned audit log entry keys. Reason: com.sap.aii.af.ra.ms.api.DeliveryException: Could not get unassigned audit log entry keys, due to: ORA-01089: immediate shutdown in progress - no operations are permitted
, Time: 01.02.2009 10:00:20)
At the adapter engine monitoring at RWB
Adapter type JPR has following error message
SLD host:port = ERPDEV02:50400
Error getting JPR configuration from SLD. Exception: No entity of class SAP_BusinessSystem for DXI.SystemHome.erpdev02 found
No access to get JPR configuration
I am getting following error when try to send a file from other business system to XI server.
<SOAP:Envelope xmlns:SOAP=''http://schemas.xmlsoap.org/soap/envelope/''><SOAP:Header/><SOAP:Body/></SOAP:Envelope>
as the response.
<?xml version='1.0'?>
<!-- see the documentation -->
<SOAP:Envelope xmlns:SOAP='http://schemas.xmlsoap.org/soap/envelope/'>
<SOAP:Body>
<SOAP:Fault>
<faultcode>SOAP:Server</faultcode>
<faultstring>Server Error</faultstring>
<detail>
<s:SystemError xmlns:s='http://sap.com/xi/WebService/xi2.0'>
<context>XIAdapter</context>
<code>ADAPTER.JAVA_EXCEPTION</code>
<text><![CDATA[
com.sap.aii.af.service.administration.api.cpa.CPAChannelStoppedException: Channel stopped by administrative task.
at com.sap.aii.af.service.administration.impl.cpa.CommonCPA.checkChannelState(CommonCPA.java:58)
at com.sap.aii.af.service.administration.impl.cpa.LookupManagerDelegate.getCPAObject(LookupManagerDelegate.java:83)
at com.sap.aii.af.mp.soap.ejb.XISOAPAdapterBean.setup(XISOAPAdapterBean.java:227)
at com.sap.aii.af.mp.soap.ejb.XISOAPAdapterBean.process(XISOAPAdapterBean.java:689)
at com.sap.aii.af.mp.module.ModuleLocalLocalObjectImpl0_3.process(ModuleLocalLocalObjectImpl0_3.java:103)
at com.sap.aii.af.mp.ejb.ModuleProcessorBean.process(ModuleProcessorBean.java:292)
at com.sap.aii.af.mp.processor.ModuleProcessorLocalLocalObjectImpl0_0.process(ModuleProcessorLocalLocalObjectImpl0_0.java:103)
at com.sap.aii.af.mp.soap.web.MessageServlet.callModuleProcessor(MessageServlet.java:174)
at com.sap.aii.af.mp.soap.web.MessageServlet.doPost(MessageServlet.java:436)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.runServlet(HttpHandlerImpl.java:401)
at com.sap.engine.services.servlets_jsp.server.HttpHandlerImpl.handleRequest(HttpHandlerImpl.java:266)
at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(RequestAnalizer.java:387)
at com.sap.engine.services.httpserver.server.RequestAnalizer.startServlet(RequestAnalizer.java:365)
at com.sap.engine.services.httpserver.server.RequestAnalizer.invokeWebContainer(RequestAnalizer.java:944)
at com.sap.engine.services.httpserver.server.RequestAnalizer.handle(RequestAnalizer.java:266)
at com.sap.engine.services.httpserver.server.Client.handle(Client.java:95)
at com.sap.engine.services.httpserver.server.Processor.request(Processor.java:175)
at com.sap.engine.core.service630.context.cluster.session.ApplicationSessionMessageListener.process(ApplicationSessionMessageListener.java:33)
at com.sap.engine.core.cluster.impl6.session.MessageRunner.run(MessageRunner.java:41)
at com.sap.engine.core.thread.impl3.ActionObject.run(ActionObject.java:37)
at java.security.AccessController.doPrivileged(Native Method)
at com.sap.engine.core.thread.impl3.SingleThread.execute(SingleThread.java:100)
at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)
]]></text>
</s:SystemError>
</detail>
</SOAP:Fault>
</SOAP:Body>
</SOAP:Envelope>
Thanks for Ur time
SenthilHi Senthil,
As suggested by others there is no problem with your communication channels ,,, the problem you face is because Your DBA has initiated the SHUTDOWN IMMEDIATE command. This will shutdown Oracle and terminate all active operations.
Please tell him to start the services and Wait until Oracle has been restarted by your DBA.
Regards
joel -
Sccheck fails with connection refused
I have a 2 node cluster that was just upgraded from SC3.0 to SC3.1u4 (8/05). sccheck passes on node aemhrsDV1 and fails with the following message on aemhrsDV2:
root@aemhrsDV2 # sccheck -b -v2 -h aemhrsDV2
sccheck: Requesting explorer data and node report from aemhrsDV2.
sccheck: aemhrsDV2 error: Connection refused
sccheck: Unable to run checks on: aemhrsDV2All SC3.1 patches have been applied. All availble agents were updated using 'scinstall -u update -s all' and re-registered. Test ip addresses were supplied during 'scinstall -u update -S interact'.
Both nodes come up fine individually in cluster (through step 10) but together the log /var/cluster/ucmm/ucmm_reconf.log shows:
Thu Aug 9 21:47:04 EST 2007 SUNWscucm.ucmm_reconf udlm started in cmmstep2
Thu Aug 9 21:48:44 EST 2007 SUNWscucm.ucmm_reconf udlm completed successfully in cmmstep2
Thu Aug 9 21:48:44 EST 2007 SUNWscucm.ucmm_reconf cvm started in cmmstep2
Thu Aug 9 21:48:45 EST 2007 SUNWscucm.ucmm_reconf cvm completed successfully in cmmstep2
Thu Aug 9 21:48:45 EST 2007 SUNWscucm.ucmm_reconf ucmm reconfiguration step 2 completed
Thu Aug 9 21:48:45 EST 2007 SUNWscucm.ucmm_reconf Step: cmmreturn CURRNODES=0 1Running scstat -i hangs forever at IPMP Groups with both nodes in cluster but separately displays the adapters and status.
Node 1
root@aemhrsDV1 # ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.100.30 netmask ffffff00 broadcast 192.168.100.255
ether 0:3:ba:39:f5:62
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.101.30 netmask ffffff00 broadcast 192.168.101.255
ce1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.251.30 netmask ffffff00 broadcast 192.168.251.255
ether 0:3:ba:39:f5:ef
ge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 10.10.3.30 netmask ffffff00 broadcast 10.10.3.255
groupname nafo0
ether 0:3:ba:b:e:8
ge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 10.10.3.31 netmask ffffff00 broadcast 10.10.3.255
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
inet 10.10.3.33 netmask ffffff00 broadcast 10.10.3.255
groupname nafo0
ether 0:3:ba:17:97:d5
qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
inet 10.10.3.32 netmask ffffff00 broadcast 10.10.3.255
qfe2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
inet 192.168.2.112 netmask ffffff00 broadcast 192.168.2.255
ether 0:3:ba:17:97:d7
qfe7: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 7
inet 172.16.1.1 netmask ffffff80 broadcast 172.16.1.127
ether 0:3:ba:17:98:68
qfe3: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 8
inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255
ether 0:3:ba:17:97:d8
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1486 index 9
inet 172.16.193.1 netmask ffffff00 broadcast 172.16.193.255
ether 0:0:0:0:0:1Node 2
root@aemhrsDV2 # ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.100.40 netmask ffffff00 broadcast 192.168.100.255
ether 0:3:ba:39:f5:50
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.101.40 netmask ffffff00 broadcast 192.168.101.255
ce1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 192.168.251.40 netmask ffffff00 broadcast 192.168.251.255
ether 0:3:ba:39:f5:20
ge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 10.10.3.40 netmask ffffff00 broadcast 10.10.3.255
groupname nafo0
ether 0:3:ba:b:41:22
ge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 10.10.3.41 netmask ffffff00 broadcast 10.10.3.255
qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5
inet 10.10.3.43 netmask ffffff00 broadcast 10.10.3.255
groupname nafo0
ether 0:3:ba:17:96:2d
qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
inet 10.10.3.42 netmask ffffff00 broadcast 10.10.3.255
qfe0:2: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 5
inet 10.10.3.35 netmask ffffff00 broadcast 10.10.3.255
qfe7: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 6
inet 172.16.1.2 netmask ffffff80 broadcast 172.16.1.127
ether 0:3:ba:17:97:18
qfe3: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 7
inet 172.16.0.130 netmask ffffff80 broadcast 172.16.0.255
ether 0:3:ba:17:96:30
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1486 index 8
inet 172.16.193.2 netmask ffffff00 broadcast 172.16.193.255
ether 0:0:0:0:0:2
root@aemhrsDV1 # scstat
-- Cluster Nodes --
Node name Status
Cluster node: aemhrsDV1 Online
Cluster node: aemhrsDV2 Online
-- Cluster Transport Paths --
Endpoint Endpoint Status
Transport path: aemhrsDV1:qfe7 aemhrsDV2:qfe7 Path online
Transport path: aemhrsDV1:qfe3 aemhrsDV2:qfe3 Path online
-- Quorum Summary --
Quorum votes possible: 3
Quorum votes needed: 2
Quorum votes present: 3
-- Quorum Votes by Node --
Node Name Present Possible Status
Node votes: aemhrsDV1 1 1 Online
Node votes: aemhrsDV2 1 1 Online
-- Quorum Votes by Device --
Device Name Present Possible Status
Device votes: /dev/did/rdsk/d3s2 1 1 Online
-- Device Group Servers --
Device Group Primary Secondary
Device group servers: rmt/1 - -
Device group servers: rmt/2 - -
-- Device Group Status --
Device Group Status
Device group status: rmt/1 Offline
Device group status: rmt/2 Offline
-- Multi-owner Device Groups --
Device Group Online Status
-- Resource Groups and Resources --
Group Name Resources
Resources: nfs-test aemhrsDVlh
Resources: hastp-aemhrsDV1-rg -
Resources: hastp-aemhrsDV2-rg -
Resources: rac-framework-rg rac_framework rac_udlm rac_cvm
-- Resource Groups --
Group Name Node Name State
Group: nfs-test aemhrsDV1 Offline
Group: nfs-test aemhrsDV2 Online
Group: hastp-aemhrsDV1-rg aemhrsDV1 Online
Group: hastp-aemhrsDV2-rg aemhrsDV2 Online
Group: rac-framework-rg aemhrsDV1 Pending online
Group: rac-framework-rg aemhrsDV2 Online
-- Resources --
Resource Name Node Name State Status Message
Resource: aemhrsDVlh aemhrsDV1 Offline Offline
Resource: aemhrsDVlh aemhrsDV2 Online Online - LogicalHostname online.
Resource: rac_framework aemhrsDV1 Starting Degraded - reconfiguration in progress
Resource: rac_framework aemhrsDV2 Online Online
Resource: rac_udlm aemhrsDV1 Offline Offline
Resource: rac_udlm aemhrsDV2 Online Online
Resource: rac_cvm aemhrsDV1 Offline Offline
Resource: rac_cvm aemhrsDV2 Online Online
-- IPMP Groups --
Node Name Group Status Adapter Status
--------- ----- ------ ------- ------I believe the reconfig is coming from aemhrsDV2 since sccheck will not run.
Suggestions?Also the output from the sccheck commad using 'ksh -x' shows where the error is occurring and exiting with a status of 107.
The line executed is 765 of the script.
# echo "CLASSPATH: ${CLASSPATH}"
${JAVA} -classpath ${CLASSPATH} ${JDEFINES_LIST} ${JAVA_CLIENT} &
JVM_PID=$!
+ JVM_PID=13881
# now wait for java to exit
wait $JVM_PID
+ wait 13881
+ /usr/java/bin/java -classpath /usr/cluster/lib/sccheck:/usr/cluster/lib/sccheck/sccheck.jar:/usr/cluster/lib/sccheck/k
ae/kae.jar:/usr/cluster/lib/sccheck/kae/kae-libs.jar:/usr/cluster/lib/sccheck/kae/eras-common.jar:/usr/cluster/lib/scche
ck/kae/common-libs.jar:/usr/cluster/lib/sccheck/kae/eras-parsers.jar:/usr/cluster/lib/sccheck/kae/explorer-input-source.
jar:/usr/cluster/lib/sccheck/kae/resources:/var/cluster/sccheck/tmp/client.0 -Dcom.sun.eras.common.logging4.config.file=
/usr/cluster/lib/sccheck/sccheck.logging4.properties -Dke.client.log=/var/cluster/logs/sccheck/ke-client.log.0 -Dke.clie
nt.xsldir=/var/cluster/sccheck/tmp/client.0 -Dsccheck.clustername=aemhrsDV -Dsccheck.clustermode=true -Dsccheck.localnam
e=aemhrsDV2 -Dsccheck.privatelocalname=clusternode2-priv -Dsccheck.publicnodenames=aemhrsDV1,aemhrsDV2 -Dsccheck.private
nodenames=clusternode1-priv,clusternode2-priv -Dsccheck.clientnumber=0 -Dsccheck.explorersgzDir=/var/cluster/sccheck/exp
lorers-gz -Dsccheck.explorersUnpackDir=/var/cluster/sccheck/explorers.0 -Dsccheck.gunzip=/usr/bin/gunzip -Dsccheck.tar=/
usr/bin/tar -Dsccheck.brief=true -Dsccheck.vverbose=true -Dsccheck.hostlist=aemhrsDV2 -Dsccheck.outputDir=/var/cluster/s
ccheck/reports.2007-08-10.09:19:37 com.sun.cluster.sccheck.SccheckClient
sccheck: Requesting explorer data and node report from aemhrsDV2.
sccheck: aemhrsDV2 error: Connection refused
sccheck: Unable to run checks on: aemhrsDV2
typeset -i exitstatus=$?
+ typeset -i exitstatus=107
# constuct closing message if reports were created
if ((${exitstatus} > 0)) && ((${exitstatus} < 100)); then
typeset severity=""
printf ""
case "${exitstatus}" in
1) severity=$(gettext '1 (LOW)');;
2) severity=$(gettext '2 (MODERATE)');;
3) severity=$(gettext '3 (HIGH)');;
4) severity=$(gettext '4 (CRITICAL)');;
esac
printf "$(gettext '%s: One or more checks failed.')\n" ${PROG}
printf "$(gettext '%s: The greatest severity of all check failures was %s.')\n" ${PROG} "${severity}"
printf "$(gettext '%s: Reports are in %s.')\n" ${PROG} ${OUTPUTDIR}
fi
+ (( 107 > 0 ))
+ (( 107 < 100 ))
cleanup
+ cleanup
exit ${exitstatus}
+ exit 107 -
As per Split brain syndrome in Oracle RAC in case of inter-connect failures the master node will evict other/dead nodes .
Let say 2 node RAC configuration node 1 is defined as master node (by some parameter like load and others) incase of network failures node 1 will terminate node 2 from cluster.
;l
what happens if master node, in this case node 1 fails. which will terminate node 1 and is node 2 will become master node ?Hi,
It occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome.
In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.
When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.
You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.
And also You can go through Metalink Note ID: 219361.1 -
Urgent help needed; Database shutdown issues.
Urgent help needed; Database shutdown issues.
Hi all,
I am trying to shutdown my SAP database and am facing the issues below, can someone please suggest how I can go about resolving this issue and restart the database?
SQL> shutdown immediate
ORA-24324: service handle not initialized
ORA-24323: value not allowed
ORA-01089: immediate shutdown in progress - no operations are permitted
SQL> shutdown abort
ORA-01031: insufficient privileges
Thanks and regards,
IqbalHi,
check SAP Note 700548 - FAQ: Oracle authorizations
also check Note 834917 - Oracle Database 10g: New database role SAPCONN
regards,
kaushal -
Cannot connect to GET STARTED page
Hello,
First off, I am sorry, I did a search and saw numerous posts on this issue, but about all of them either ended their replies after a user asked "Did you do this or that ?" or they just stopped posting... at most I saw some topics where it magically worked later on, but no real step-by-step instructions.
So, the problem: I installed XE 11g, no apparent problems during install (no warnings so I assume everything is OK), tried the GET STARTED link, but it said it can't connect.
System: Win XP PRO SP3, 2 gb ram using Firefox 7.0.1
DB: Oracle 11g Express Edition
Antivirus: avast 6 (firewall included), but disabled during testing...
lsnrctl:
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for 32-bit Windows: Version 11.2.0.2.0 - Produ
ction
Start Date 05-OCT-2011 02:36:42
Uptime 0 days 0 hr. 21 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Default Service XE
Listener Parameter File E:\Database\Oracle\app\oracle\product\11.2.0\server\network\admin\listener.ora
Listener Log File E:\Database\Oracle\app\oracle\diag\tnslsnr\EASTFRONTIERS03\listener\alert\log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\\.\pipe\EXTPROC1ipc)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=EASTFRONTIERS03)(PORT=1521)))
Services Summary...
Service "CLRExtProc" has 1 instance(s).
Instance "CLRExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "xe" has 1 instance(s).
Instance "xe", status BLOCKED, has 1 handler(s) for this service...
The command completed successfullytns:
XE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = XE)
EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
ORACLR_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(CONNECT_DATA =
(SID = CLRExtProc)
(PRESENTATION = RO)
) listener:
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = E:\Database\oracle\app\oracle\product\11.2.0\server)
(PROGRAM = extproc)
(SID_DESC =
(SID_NAME = CLRExtProc)
(ORACLE_HOME = E:\Database\oracle\app\oracle\product\11.2.0\server)
(PROGRAM = extproc)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521))
DEFAULT_SERVICE_LISTENER = (XE)Services:
OracleJobSchedulerXE
OracleMTSRecoveryService
OracleServiceXE (started)
OracleXEClrAgent
OracleXETNSListener (started)sqlplus:
SQL> connect
Enter user-name: SYS
Enter password:
ERROR:
ORA-01089: immediate shutdown in progress - no operations are permitted
Process ID: 0
Session ID: 0 Serial number: 0alert_xe:
Wed Oct 05 02:37:35 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit record
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - Production.
Using parameter settings in client-side pfile E:\DATABASE\ORACLE\APP\ORACLE\PRODUCT\11.2.0\SERVER\CONFIG\SCRIPTS\INIT.ORA on machine EASTFRONTIERS03
System parameters with non-default values:
sessions = 172
memory_target = 760M
control_files = "E:\DATABASE\ORACLE\APP\ORACLE\ORADATA\XE\CONTROL.DBF"
compatible = "11.2.0.0.0"
db_recovery_file_dest = "E:\Database\Oracle\app\oracle\fast_recovery_area"
db_recovery_file_dest_size= 10G
undo_management = "AUTO"
undo_tablespace = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
dispatchers = "(PROTOCOL=TCP) (SERVICE=XEXDB)"
shared_servers = 4
job_queue_processes = 4
audit_file_dest = "E:\DATABASE\ORACLE\APP\ORACLE\ADMIN\XE\ADUMP"
db_name = "XE"
open_cursors = 300
diagnostic_dest = "E:\DATABASE\ORACLE\APP\ORACLE"
Wed Oct 05 02:37:53 2011
PMON started with pid=2, OS id=2308
Wed Oct 05 02:37:53 2011
PSP0 started with pid=3, OS id=1168
Wed Oct 05 02:37:54 2011
VKTM started with pid=4, OS id=1920 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Wed Oct 05 02:37:54 2011
GEN0 started with pid=5, OS id=372
Wed Oct 05 02:37:54 2011
DIAG started with pid=6, OS id=1580
Wed Oct 05 02:37:54 2011
DBRM started with pid=7, OS id=1552
Wed Oct 05 02:37:54 2011
DIA0 started with pid=8, OS id=2668
Wed Oct 05 02:37:54 2011
MMAN started with pid=9, OS id=2132
Wed Oct 05 02:37:54 2011
DBW0 started with pid=10, OS id=3716
Wed Oct 05 02:37:54 2011
LGWR started with pid=11, OS id=3952
Wed Oct 05 02:37:54 2011
CKPT started with pid=12, OS id=2992
Wed Oct 05 02:37:54 2011
SMON started with pid=13, OS id=2616
Wed Oct 05 02:37:54 2011
RECO started with pid=14, OS id=1764
Wed Oct 05 02:37:54 2011
MMON started with pid=15, OS id=4012
Wed Oct 05 02:37:54 2011
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 4 shared server(s) ...
ORACLE_BASE from environment = E:\Database\Oracle\app\oracle
Wed Oct 05 02:37:55 2011
AUD: OS Error = 1717 encountered while writing audit record
Wed Oct 05 02:37:54 2011
MMNL started with pid=16, OS id=1516
Wed Oct 05 02:37:57 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit record
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
License high water mark = 1
All dispatchers and shared servers shutdown
AUD: OS Error = 1717 encountered while writing audit record
Starting ORACLE instance (normal)
Wed Oct 05 02:38:02 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit record
Wed Oct 05 02:38:03 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit record
Wed Oct 05 02:38:03 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit record
Wed Oct 05 02:38:03 2011
AUD: OS Error = 1717 encountered while writing audit record
AUD: OS Error = 1717 encountered while writing audit recordFrom what I could gather from other posts here (and on other sites) the problem is I don't have anything listening on 8080. Other than that, all seems OK (I think).
So, can anyone tell me\help me on why that one isn't starting ? I reinstalled this 6 times and still can't figure out the problem... (as you saw, I even replaced the tns hostname with 127.0.0.1 out of paranoia). It's seriously driving me mad. Every topic I searched for (from the problem itself to even a step-by-step for dummies) shows an install, then the interface working... I see some questions regarding "I can't connect to..." but most are unanswered :(
Thanks in advance!
P.S. let me know if I forgot to mention anything relevantTheNomad wrote:
orafad: Sad to say that was the full output...
C:\Documents and Settings\User>sqlplus / as SYSDBA
SQL*Plus: Release 11.2.0.2.0 Production on Thu Oct 6 12:55:47 2011
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> select 1 from dual
2 /
select 1 from dual
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL>
jgarry: I believe what you explained may be related to the fact some reported the DB doesn't work properly when the event log is full (I just emptied mine so it doesn't apply). Normally I disable it but without it started, the DB is not working properly (not that I can get it started anyway).
There was a post on a site I found hinting at 3 things:
1. ORACLE_HOME must be defined as an env-var
2. ORACLE_SID must be defined as an env-var
3. there might be a problem with spfile
Regarding those:
1. shouldn't the setup declare it automatically ? I don't have it declared, but I added it (no change)As Orafad pointed out, Windows is different. It does have a search hierarchy it follows to figure out what the environment should be. So first, unset what you set and go into a cmd window and:
sqlplus sys as sysdba
Edit: Oops, I meant
sqlplus / as sysdba
startup
2. same as above, except I am not sure what SID to add so I didn't declare it ... The only 2 I have are PLSExtProc and CLRExtProc in tnsnames by default. I tried with each but the only thing that changes is instead of getting the shared memory realm error I get a TNS adapter error... trading one error for another is not funFor XE, the ORACLE_SID is XE (unless you do screwy things to change it). So now that I see you had some other db installed before, I suspect an environment collision. Delve into your registry and see what oracle definitions you have - this is a lot of work, things are spread around some, but searching for oracle is usually a good start. I'm not up on the 11.2 details (or others, really, I just have vague memories of seeing MOS notes that purported to explain clean up entirely, without truly doing so, and I avoid Windows as much as possible). I would uninstall XE, uninstall the other one, then search the registry for any leftover stuff, then install XE and see if it all magically works. But that is a fairly brain-dead way to deal with it.
So that said, you can set up a little batch file to set your environment in a cmd window (specifically ORACLE_HOME and ORACLE_SID), and go into sqlplus / as sysdba and give the startup command. If you get the memory realm error, you have one of those wrong, if you get the tnsnames errors you can fix those following the advice in Ed Steven's blog. You can run tnsping to see which tnsnames.ora you are using, if you use one (which is determined by an @ sign in your connection string, which you should not need to connect locally, but if you do, you need to have XE defined in the tnsnames.ora, and may need to set TNS_ADMIN to get the right one).
Your tnsnames.ora should have an entry like this:
XE =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = Yourcomputer.yourdomian.whatever)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = XE)
)You might want to search for any tnsname.ora on your computer, the default correct one would be: C:\oraclexe\app\oracle\product\11.2.0\server\network\ADMIN\tnsnames.ora
3. I don't have that file in the folder but I suspect it might be created on a first successful launch of the DBWithout an spfile you need to use a pfile, then create a pfile from spfile. But there should be one at C:\oraclexe\app\oracle\product\11.2.0\server\dbs\SPFILEXE.ORA (if you used default installation).
>
Other than that I didn't find anything helpful in logs (or don't know where else to look), nor on sites.
I did have WAMP installed at a point (uninstalled it) but if it were to conflict I guess SETUP would tell me (since the XE license doesn't let you use 2 DBs at the same time)Is that some mysql thing? That shouldn't interfere. It's only installing multiple Oracle products related some what to the Oracle database that can be a problem, and normally isn't with the more recent stuff.
I swear to god this is killing my neurons :(When you find the solution, you will slap your forehead.
Edited by: jgarry on Oct 6, 2011 9:36 AM -
Hi all,
I am new to XML Publisher.I have tried to publish a Oracle Report in EBS with XML publisher, but failed.
I defined the data definition and the Template as said in the UserGuide.Then i set up the report (in the Concurrent..Program..Define window) to generate a XML data.
I even defined the default template in the Update Concurrent Manager Window but when i go the the standard Concurrent Request Submission window i dont see the default Template in the Upon Completion..Layout window.
After this if i submit the concurrent Request it gets completed with warnings,"Internal error Unable to start Oracle toolkit.".On hitting View output tab i get the alert----- No Output generated for this request.
I cannot understand where i am going wrong.
Any help wold be great as i am new to XMLP:
I have done the installation setup test by invoking the fnd_installation.get_app_info function and it returned me the 'I' status.So i just hope may be the installation setup is fine.Hi,
when i open the Opp log file to see the error for case 2.i found the following error in log file.
Template code: FNDCPPGD_XML
Template app: FND
Language: en
Territory: US
Output type: PDF
[3/21/07 8:00:41 PM] [UNEXPECTED] [95374:RT2117499] java.sql.SQLException: No corresponding LOB data found :SELECT L.FILE_DATA FILE_DATA, LENGTH(L.FILE_DATA) FILE_LENGTH, L.LANGUAGE LANGUAGE, L.TERRITORY TERRITORY, B.DEFAULT_LANGUAGE DEFAULT_LANGUAGE, B.DEFAULT_TERRITORY DEFAULT_TERRITORY,B.TEMPLATE_TYPE_CODE TEMPLATE_TYPE_CODE, B.USE_ALIAS_TABLE USE_ALIAS_TABLE, B.START_DATE START_DATE, B.END_DATE END_DATE, B.TEMPLATE_STATUS TEMPLATE_STATUS, B.USE_ALIAS_TABLE USE_ALIAS_TABLE FROM XDO_LOBS L, XDO_TEMPLATES_B B WHERE L.LOB_TYPE = :1 AND L.APPLICATION_SHORT_NAME= :2 AND L.LOB_CODE = :3 AND L.APPLICATION_SHORT_NAME = B.APPLICATION_SHORT_NAME AND L.LOB_CODE = B.TEMPLATE_CODE AND ( (L.LANGUAGE = :4 AND L.TERRITORY = :5 ) OR (L.LANGUAGE = :6 AND L.TERRITORY = :7) OR (L.LANGUAGE= B.DEFAULT_LANGUAGE AND L.TERRITORY= B.DEFAULT_TERRITORY ) )
at oracle.apps.xdo.oa.schema.server.TemplateInputStream.initStream(TemplateInputStream.java:318)
at oracle.apps.xdo.oa.schema.server.TemplateInputStream.<init>(TemplateInputStream.java:183)
at oracle.apps.xdo.oa.schema.server.TemplateHelper.getTemplateFile(TemplateHelper.java:776)
at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:1269)
at oracle.apps.fnd.cp.opp.XMLPublisherProcessor.process(XMLPublisherProcessor.java:229)
at oracle.apps.fnd.cp.opp.OPPRequestThread.run(OPPRequestThread.java:153)
[3/21/07 8:00:41 PM] [95374:RT2117499] Completed post-processing actions for request 2117499.
[3/21/07 8:41:50 PM] [OPPServiceThread0] Post-processing request 2117501.
[3/21/07 8:41:50 PM] [95374:RT2117501] Executing post-processing actions for request 2117501.
[3/21/07 8:41:50 PM] [95374:RT2117501] Starting XML Publisher post-processing action.
[3/21/07 8:41:50 PM] [95374:RT2117501]
Template code: FNDCPPGD_XML
Template app: FND
Language: en
Territory: US
Output type: PDF
[3/21/07 8:41:50 PM] [UNEXPECTED] [95374:RT2117501] java.sql.SQLException: No corresponding LOB data found :SELECT L.FILE_DATA FILE_DATA, LENGTH(L.FILE_DATA) FILE_LENGTH, L.LANGUAGE LANGUAGE, L.TERRITORY TERRITORY, B.DEFAULT_LANGUAGE DEFAULT_LANGUAGE, B.DEFAULT_TERRITORY DEFAULT_TERRITORY,B.TEMPLATE_TYPE_CODE TEMPLATE_TYPE_CODE, B.USE_ALIAS_TABLE USE_ALIAS_TABLE, B.START_DATE START_DATE, B.END_DATE END_DATE, B.TEMPLATE_STATUS TEMPLATE_STATUS, B.USE_ALIAS_TABLE USE_ALIAS_TABLE FROM XDO_LOBS L, XDO_TEMPLATES_B B WHERE L.LOB_TYPE = :1 AND L.APPLICATION_SHORT_NAME= :2 AND L.LOB_CODE = :3 AND L.APPLICATION_SHORT_NAME = B.APPLICATION_SHORT_NAME AND L.LOB_CODE = B.TEMPLATE_CODE AND ( (L.LANGUAGE = :4 AND L.TERRITORY = :5 ) OR (L.LANGUAGE = :6 AND L.TERRITORY = :7) OR (L.LANGUAGE= B.DEFAULT_LANGUAGE AND L.TERRITORY= B.DEFAULT_TERRITORY ) )
at oracle.apps.xdo.oa.schema.server.TemplateInputStream.initStream(TemplateInputStream.java:318)
at oracle.apps.xdo.oa.schema.server.TemplateInputStream.<init>(TemplateInputStream.java:183)
at oracle.apps.xdo.oa.schema.server.TemplateHelper.getTemplateFile(TemplateHelper.java:776)
at oracle.apps.xdo.oa.schema.server.TemplateHelper.processTemplate(TemplateHelper.java:1269)
at oracle.apps.fnd.cp.opp.XMLPublisherProcessor.process(XMLPublisherProcessor.java:229)
at oracle.apps.fnd.cp.opp.OPPRequestThread.run(OPPRequestThread.java:153)
[3/21/07 8:41:50 PM] [95374:RT2117501] Completed post-processing actions for request 2117501.
[3/21/07 10:15:55 PM] [GSMServiceController:95374] Received shutdown request.
[3/21/07 10:15:55 PM] [GSMServiceController:95374] Preparing to shutdown service.
[3/21/07 10:15:55 PM] [GSMServiceController:95374] Stopping all Service Threads.
[3/21/07 10:15:55 PM] [OPPServiceThread0] Preparing to shut down service thread.
[3/21/07 10:15:55 PM] [OPPServiceThread1] Preparing to shut down service thread.
[3/21/07 10:17:13 PM] [EXCEPTION] [OPPAQMON:95374] ORA-01089: immediate shutdown in progress - no operations are permitted
[3/21/07 10:17:13 PM] [UNEXPECTED] [OPPAQMON:95374] java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:589)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1972)
at oracle.jdbc.ttc7.TTC7Protocol.executeFetch(TTC7Protocol.java:977)
at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:2199)
at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:2059)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2976)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:656)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:734)
at oracle.apps.fnd.cp.opp.OPPAQMonitor.waitForMessage(OPPAQMonitor.java:412)
at oracle.apps.fnd.cp.opp.OPPAQMonitor.run(OPPAQMonitor.java:110)
at java.lang.Thread.run(Thread.java:534)
[3/21/07 10:17:25 PM] [UNEXPECTED] [OPPServiceThread0] java.sql.SQLException: ORA-01089: immediate shutdown in progress - no operations are permitted
at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:589)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1972)
at oracle.jdbc.ttc7.TTC7Protocol.executeFetch(TTC7Protocol.java:977)
at oracle.jdbc.dbaccess.DBAccess.executeFetchNeedDefines(DBAccess.java:283)
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2685)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2950)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:656)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:582)
at oracle.apps.fnd.cp.opp.OPPServiceThread.okToShutdown(OPPServiceThread.java:306)
at oracle.apps.fnd.cp.opp.OPPServiceThread.mainLoop(OPPServiceThread.java:146)
at oracle.apps.fnd.cp.gsf.BaseServiceThread.run(BaseServiceThread.java:125)
Maybe you are looking for
-
How can you get just an iPad 1? i cant find it on this site?
How can i purchase just an iPad 1 from this site? cant find it anywhere?
-
Document Splitting - full balance sheet
Hello, Do we we have to setup Document splitting functionality when the requirement is to have a full balance sheet by Segment, Profit Center or by Business Area or are there other possiblities to meet this requirement? Many thanks
-
i reinstall my wondows and itunes but now when i want to be connect to my ipad ask me if i sync will remove all the information i installed already on it .culd some one help me in this regards,i already copied all the foldar i prevousely had for itun
-
I am trying to update my Iphone 4 to iSO 4.3 but when I following the instruction under Setting - General there is no option for Software udpate. Do I need to do this another way?
-
I was asking for basic setup information for webdav a while back and got some helpful information (mostly the pointer to the Web Tech Admin guide which I had somehow not seen). I have now set up several realms, one for each afp share that I have set