Cluster / ASM restore

All,
I've just come into a place and they have an issue in that i need to fix. Basically they have had disk corruption and have lost all the binaries etc for the Cluster and ASM ie everything under /u01/asm and /u01/crs on one of the nodes. They have an RMAN backup of the database (which is still running happily on the other node), but that's all. Am i correct in thinking that the easiest / quickest way to fix this would be to drop the faulty node and then re-add it etc or is there a better way ?
Oracle 10.2.0.4, AIX 6.1, 2 node RAC cluster.
Many thanks,
graham

Easiest way is to restore from backup.
If you do not have a backup, then go through the process to remove the node from inventory, CRS etc, then go through the process to add the node back to the cluster.

Similar Messages

Non-asm to asm restore using duplicate

oracle version 11.2.0.2.0
hello guys a quick question, I am trying to restore a database from non-asm to asm. I want to know if the following will work with the duplicate database.
SET NEWNAME for DATAFILE '/clu15/oracle/database/PROD/dbf/NDX_PROD.dbf' to '+DGDEV1DBDATA03/database/DEV/dbf/NDX_PROD.dbf';
SET NEWNAME for DATAFILE '/clu15/oracle/database/PROD/dbf/NDX_PROD_01.dbf' to '+DGDEV1DBDATA03/database/DEV/dbf/NDX_PROD_01.dbf';
SET NEWNAME for DATAFILE '/clu15/oracle/database/PROD/dbf/NDX_PROD_02.dbf' to '+DGDEV1DBDATA03/database/DEV/dbf/NDX_PROD_02.dbf';
SET NEWNAME for DATAFILE '/clu15/oracle/database/PROD/dbf/NDX_PROD_03.dbf' to '+DGDEV1DBDATA03/database/DEV/dbf/NDX_PROD_03.dbf';
Thank You

This above will work.
If you want to use OFA architecture, you can just give
SET NEWNAME for DATAFILE '/clu15/oracle/database/PROD/dbf/NDX_PROD.dbf' to '+DGDEV1DBDATA03';
You can also set the parameter db_file_name_convert and log_file_name_convert.

ASM restore

11g ASM RAC, whole data raw disk group lost as SA mount raw disks to anther file system,
rebuild +data raw disk group.
rebuild spfile, restore controlfile.
do rman restore database
RMAN> restore database;
Starting restore at 02-OCT-08
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +DATA/tester/datafile/system.260.662163575
channel ORA_DISK_1: restoring datafile 00002 to +DATA/tester/datafile/sysaux.266.662163575
channel ORA_DISK_1: restoring datafile 00003 to +DATA/tester/datafile/undotbs1.264.662163577
channel ORA_DISK_1: restoring datafile 00004 to +DATA/tester/datafile/users.263.662163577
channel ORA_DISK_1: restoring datafile 00005 to +DATA/tester/datafile/example.258.662163761
channel ORA_DISK_1: restoring datafile 00006 to +DATA/tester/datafile/undotbs2.256.662163961
channel ORA_DISK_1: reading from backup piece +DG1/tester/backupset/2008_08_15/nnndf0_tag20080815t112016_0.265.662815237
channel ORA_DISK_1: ORA-19870: error while restoring backup piece +DG1/tester/backupset/2008_08_15/nnndf0_tag20080815t112016_0.265.662815237
ORA-19504: failed to create file "+DATA"
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15041: diskgroup space exhausted
I have 3 20gb raw disk +data group, tester is a small sample db<1gb,
ASMCMD> cd data
ASMCMD> du
Used_MB Mirror_used_MB
4407 4407
Q1. where did space go?
ASMCMD> ls -l
Type Redund Striped Time Sys Name
N CONTROLFILE/
N datafile/
N spfiletester.ora => +DATA/DB_UNKNOWN/PARAMETERFILE/SPFILE.327.6670637
Q2. the directory I manually created with " N " sign on it under sys column, while dbca created dir had a " Y" sign, what is step to restore db to asm storage?
Thanks.

'N' means these are user-created, but not system-generated ASM entries. Normally that happens when directories are created via 'alter diskgroup add directory ...' or mkdir statement inside asmcmd. User-created entries are aliases and ASM cannot resolve filenames to system names like '+DATA/tester/datafile/system.260.662163575'. Exactly that RMAN wants to do. Because you lost all your metadata you have to recreate your whole ASM environment (by the way 11g makes it possible to backup the metadata itself,which makes restore/recovery much easier). Follow the instructions from this metalink note:
Steps to Re-Create ASM Diskgroups
Doc ID: NOTE:268481.1
Werner

Recreating ASM & Restoring database

I have an Oracle 10g R2 server with 2 disk volumes. 1st one is for the OS and the Oracle install and the 2nd one is for the database managed by Oracle ASM. I have a full backup of the database I took using OEM.
I'm having to rebuild the 2nd disk volume therefore loosing everything on it, but will still have the Oracle install unaffected.
My question is after rebuilding the disk would it be OK to just create a new ASM group and create a new database. Then just restore the backup on to it using OEM?
Thanks for your help.

We're restoring a database from tape using Oracle 10.2.0.5. We explcitely specify the location for every single file with SET NEWNAME FOR DATAFILE X. The files that get restored don't match the NEWNAME format we set it to even though the log says it restored the file to that name. I don't know what's occuring under the hood of ASM, but it appears OMF is coming into play and generating names for the files. it's becoming a problem because when we restart the restore (which occurs often due to size of the database, length of time it takes, and drives going down) we start all over because RMAN doesn't know of the existing files. This doesn't seem to be normal behavior, and I know it's not like this during file based restores.
Does anyone know what's going on within ASM itself? It appears to be normal behavior when restoring to ASM and the names are later set appropriately after the SWITCH DATAFILE ALL. That's what I think would happen. It's currently a problem restarting restores.Have you enabled OMF? as db_create_file_dest = '+data'
then if you restore database from tape/disk, it will place all the datafiles as '+data/db_unique_name/datafiles';

Sun Cluster Server restore

Recently one of cluster server crashed without backup. I had to reinstall all of Solaris 8 and software such as veritas volume manager and oracle, Sun Cluster 3.0. Unfortunately, it is said that if one of both cluster servers was down, I have to reinstall both machines. Is it true?
Is there anyone can give me some advices how to fix this problem properly?
Thxs

I think I am having same problem as you before. If possible, could you please give me some instructions. I am just a beginner of Sun Cluster Server. According to your sugguestion, it looks so complex that I can handle. Presently, I have installed volumn manager and oracle on the broken node. Can you instruct me what's next? The more details the better.
Thanks in advance.

Restore backupsets from tape.

Hi,
We are on a 2 node RAC cluster, ASM. We run backups through rman.
Scenerio:
Rman backup runs and create backupsets in ASM diskgroup. We again run a tape backup to backup the backupsets to tape and delete the backupset from disk.
My question:
How can i restore the backupsets back to disk, if i have to restore a database?
I use the following to backup the backupsets to tape:
backup backupset completed before 'sysdate-2' delete input
But i'm not sure of what command to use to restore the backupsets back from tape to disk if i need to restore the database? Any help appriciated.
Thanks
Karthik

Hi,
You are keeping latest backups on disk and in case of any Disk failure, when you run RESTORE/RECOVER it will use the backups from disk.
In case if RMAN need more files then automatically restores needed archived logs from tape to disk during recovery.
Please keep backups in a separate ASM diskgroup, which should not contain active database files such as datafiles, control files, and online redo logs.
Regards,
Satishbabu Gunukula
http://oracleracexpert.blogspot.com

CloudControl 12c shows "Agent Unreachable" for Cluster Database

Hello ,
when clicking on "Targets -> Databases" our RAC database shows up as follows:
Name Type Status
PROD02 Cluster Database symbol for "Agent Unreachable"
PROD021 Database Instance symbol "green arrow" for "Up"
PROD022 Database Instance symbol "green arrow" for "Up"
I don't understand why the symbol for the Cluster Database is not a green arrow, too. The agent on each node is up and running - and when either clicking on "PROD02" or "PROD021" (or PROD022) Cloud Control offers the full functionality. Therefore the status "Agent Unreachable" is missleading (from my point of view). I don't want to remove the complete Cluster Database and discover it once again - several jobs have already been defined and I don't want to do that once again.
Did anybody else experience the same behaviour?
Any help will get appreciated...
Rgds
JH

... seems like if I'm hitting bug "BUG 13865299".
=> "12c Cluster database or Cluster ASM in Pending or "Agent unreachable" for undetermined time [ID 1463196.1]"

Installed Grid 11r2, after relink ASM won't start on both node

Hi ,
after a succesfully installation of grid 11 r2 on hp-ux itanium, i'm facing a problem:
scenario:
the cluster is online with no errors.
I have to relink binaries because some failed during installation.
Unlocked grid home
relink, all binaries are now ok.
Lock grid home and start crs.
This is what i've done.
Now if i start on node 1 the ASM instance i recieve:
ORA-15150: instance lock mode ‘EXCLUSIVE’ conflicts with other ASM instance(s)
CAUSE:
Some other ASM instance used the lock name space in a conflicting mode.
ACTION:
Shut down the other instance or start up in compatible mode. Alternatively, set the DB_UNIQUE_NAME initialization parameter to avoid the conflict.
I restarted the crs on both nodes and the error is still here.
With crs stopped there are no Oracle processes.
Someone know what is compatible mode startup?
I've also tried to unregister from cluster ASM instance and re-add but the error still raise.
Any Idea?
Thanks Roberto

I'm the writer of the post.
Solution found:
first to relink all the oracle grid i forgot to do first:
make -f ins_rdbms.mk rac_on

11.2 ASM and IBM HACMP

Hi,
We are looking to upgrade our Oracle database server cluster from 10.2.0.3 to 11.2 this year. The 10.2.0.3 setup currently uses filesystems/datafiles and IBM's HACMP fails over to backup node.
I would like to convert everything to ASM but we don't want to purchase RAC. If I install the 11.2 Grid Infrastructure for a Standalone Server will I be able to failover ASM to the backup node via IBM HACMP?
Thanks
Tom

Hi Tom,
Oracle Clusterware and ASM are free when used with an Oracle Database regardless of RAC. In other terms, you do not need to purchase a RAC option to use Oracle Clusterware or ASM free of charge. Hence, if you want to have a failover cluster to manage your DB in a failover manner, you could use Oracle Clusterware to do that. On http://otn.oracle.com/clusterware you will find more information regarding this setup.
The benefit for you would be: Once you have Oracle Clusterware, you can cluster ASM (all free of charge) and you would not need to fail it over using HACMP anymore. The professional version of this configuration is called Oracle RAC One node, available with 11.2. But for earlier versions, this setup might do. Note the support situation for this configuration, if you do not use RAC One Node as described, in Support Note 790189.1 You could even get rid of HACMP on these machines, since you plan on using ASM anyways.
If you choose to use HACMP to fail over ASM, you need to ask IBM whether they have a solution (agent) for that.
Hope that helps. Thanks,
Markus

ASM Device Blocks from DISK GROUP

Hi Experts,
Need your expertise in finding the issues.
Currently we are in Linux RHEL 6, oracle 11.2.0.3
We dont have oracleasm binaries but using udev.
We have 12 ASM DISK groups with names ASM_DISK_01... ASM_DISK_12
All I am interested is how can I see where the block devices are mapped .
i.e ASM_DISK_01 is mapped to /dev/sd1 on the host... like that which file i need to check.
I tried /dev/mapper/*permissiom* file ...but no luck .
Please help

Oracle is not supporting Red Hat Linux 6.x with the asmLib files. It is supporting the Oracle version of Linux, however it is no longer supplying the asmlib disk management utilities as it did in the Red Hat Linux 5.x release. You really don't need the asmlib because Red Hat Linux is provides the disk/device management with udev.
I have just successfully implemented Red Hat Linux 6.3 using Oracle Grid Infrastructure 11.2.0.3 ASM and Oracle Real Application Cluster (RAC) release 11.2.0.3 and it working like a charm. At first I thought that I had to do a lot of things differently and thinking that it was time as we know it has changed (mayan calendar end) however we are just in a different time and Oracle 11gR2 (11.2.0.3) Grid Infrastructure and Red Hat Linux 6.3 are a great compliment.
Oracle continues to improve on its Grid Infrastructure ASM and Real Application Cluster ASM as Red Hat Linux continues to provide improved functionality and enterprise management.
I'm preparing to post my most recent build.
Edited by: yakub21 on Feb 13, 2013 6:14 PM

Upgrade to 11R2 Win Cluster Only?

Hi,
i have seen with new version 11R2 the approch of the installation have change. Now the cluster is part of the Grid upgrade package.
My rac installation is a clusteware 1.1.0.7 and my rdbms is a 10.2.0.3 under Window 2003 sp2.
I there a way i can upgrade my cluster in these step (like in older version)
1- Upgrade cluster only (or cluster and asm)
2- Upgrade Database and ASM
3- Upgrade GRID at the end (because i would like to upgrade all my GRID agents and server at the same time).
I have check the installation doc. and i did not fine a way to do these step in the order i specify.
Thanks

download the document you indicate me in the answer.
Great document, but i still have my question unanswered after i bass thru.
Every time i try to upgrade my cluster the OUI force me to upgrade my GRID (or new installation) before i can upgrade my cluster, ASM and RDBMS.
When i do the step in the OUI, at one point the SCAN ask me for a 'TCP/IP host name lookup', message INS-40718.
Is there a way i can bypass this GRID upgrade (or installation) and go directly to the cluster and ASM upgrade?
If no and i understand the installation documentation, i will have to create 1 new adress for my 3 nodes in the DNS (one alias adress in DNS round robin for the 3 vip adress of my 3 nodes) and added to my HOST file, so the installation can solve it? (at this ponit i'm not shure if the 3 ip adress have to be the 3 vip or 3 new ip adress)
Can you indicate me a example what should i put in my host file for the scan client adress
example:
# public
host1 10.200.0.11
host2 10.200.0.12
host3 10.200.0.13
#vip
host1-vip 10.200.0.21
host2-vip 10.200.0.22
host3-vip 10.200.0.23
#vip
host1-priv 10.200.0.31
host2-priv 10.200.0.32
host3-priv 10.200.0.33
# scna client
host_scan ????????
Thanks
Edited by: Ron_B on 2010-04-13 13:35
Edited by: Ron_B on 2010-04-14 05:46

Plan to move database to cluster

Hi Experts,
We are in need to move to database " venkat" to the cluster without having to re-initialize it's secondaries in log shipping.
What we plan to do is--- we can Log Ship from "Venkat" database to the new cluster, take a tail log backup, apply to the cluster (and bring online), then
rebuild the log shipping configuration, taking a new log bakcup from the cluster and restoring to each of the secondaries.
Please advice me is there any other plan we can perform this task.
Thanks in advance.

Hello Venkatesh, I tired in this in my test envirnoment. below is what i think would work. my test scenerio : Database - TESTLOGSHIP Database Server : Mercury - Primary Logshipping ; Database Server : Venus - Secondary Logshipping there is no witness server
configured.
Now, I need to migrate TestLogship database to new database server 'EARTH' and configure logshipping s primary. it does not matter whether the database server 'EARTH' is clustered or not, it will be the same procedure. Assuming, your current logshipping
works fine without any issues.
1. Take full backup on the mercury server for 'testlogship ' database.
2. restore it on the 'earth' database server in recovering mode.
3. run select * from msdb.dbo.restorehistory where destination_database_name='testlogship' on the venus database server, to find which logbackup was applied.
4. make sure the none of the log shipping jobs(backup\copy\restore) are running and disable them.
5. if there any log backups taken but not applied on to venus database server - apply any unapplied log backups to the secondary logshipping database(database server : venus).
6. now, find if there are any log backups taken after the full backup(taken in step1) was taken and restore them on database server Earth in 'recovering' mode.
7. make sure both the Earth and Venus, database servers have same log backups applied. you can verify this by checking 'select * from msdb.dbo.restorehistory where destination_database_name='testlogship'
8. now, restore the database on earth with recovery.
9. right click on the database 'testlogship' on the EARTH server and --tasks -- ship transaction logs and configure logshipping, like you normal would but choose 'secondary database is initialized option' and everything else would obvious, if you have configured
log shipping before.
So, i think biggest factor is, making sure both the new primary and secondary databases are on the same log lsn before configuring the log shipping again.
EDIT: As I think about it, you can also use differnetial backups, to sync the data and lsn. i.e if there are too many log backups to be applied, just take differential and apply, this will syn the data and lsn's and i think this is some what less hassle,
instead of checking all the log backups and track which one is applied and all that..
Hope it Helps!!

Change cluster node's hostname

Just install a 4 node RAC cluster with 11gR2 infrastructure (cluster + asm), 10g RAC and 11g RAC database. Then customer told us to change the cluster node IP and hostname.
According to metalink (220970.1), it requires to reinstall the cluster.
Wondering is there any alternate way to rename cluster node's IP/hostname?

Hi user5544520,
as far as I know it is not supported to change your hostname with CRS.
You will have to remove the node from the cluster.
Change hostname and do other necessary changes.
Add node to the cluster.
There are lots of documentation available.
Look at the following:
http://www.pythian.com/news/482/changing-hostnames-in-oracle-rac/In above mentioned blog they say:
If you are brave enough, there is another way to do this.
It’s not described anywhere on the Metalink, but there are no major hacks needed to implement it.
The idea is to simply re-run the configuration of CRS (including re-formatting OCR and voting disks) and re-create the CRS resources after that.
Obviously, this is not an online operation and the whole cluster is down for the duration of rename.Hope this helps.
Edited by: user11150436 on Oct 29, 2010 4:09 PM

Oracle DBA Jobs

Hello All,
Please don't mind my question if I have violated the purpose of the forum and if so then I am really sorry.
I have been working in Oracle Database Administration since almost 3.4 years. I have mostly worked on Dev-Test Platforms where we maintain zero downtime. I do not have any hands on RAC or much knowledge . I have worked on 10g,11g.
I know the concepts and methods of implementing Data guard (standby), I have worked on RMAN(duplication,incremental backup,complete/incomplete recovery and etc),expdp/impdp and exp/imp, little amount of scripting (SQL/Shell), basic Server management(mostly have exp on Linux), Installation of RDBMS/ASM, Creation of Databases(manually/DBCA), Security(privs,roles,auditing), Tablespace and other logical structure management, HOT backup,Cold backup(and restoration from it), Database upgradation,patching(PSU, one-off,inventory attach/detach), Tracing, SQL Loader, standby monitoring and creation, Database Monitoring and maintenance, segment management and defragmentation, Database object maintenance, Configuration of OEM etc, handling different types of Oracle errors including critical errors like 600,7445 etc... Reading and extracting AWR/ASH..
Can I get a job outside of my present company? As I am not seeing much growth there.. Not even getting to have exposure to other platforms.. Most of the companies want RAC personal.
Please suggest and help me (if possible).. I am really in tension about what will be there in my future as here I will never get to work in RAC and high availibility architecture ..
That is the reason now I am thinking to switch my company for better technological growth..
What areas should I know in this phase of my career. pls suggest.. I am from INDIA
-Regards
Struggling DBA

Whether you will get another job is an entirely local matter. Ask a local recruiter how the job market is for DBAs. In my local area (in the US), it seems like there is a decent amount of demand for DBAs, but you might be working in a local economy that is stagnant.
Chances are, in 2013, that any new DBA job will involve some element of database redundancy.
Within a few years, I suspect that most places will have either RAC or Exadata systems for their critical databases. It'll probably take a significant outage before most companies will spend the cash for clustered systems, but they will eventually move that way. Even Oracle's Database Machine is a RAC configuration if you think about it - and that's what they're trying to pitch to the smaller customers.
I remember that about 10 years ago, RAC was rare (I remember installing RAC 9.2.0.1 for a client - boy, that was DIFFICULT). 'All' that you had to do to be a DBA then, was understand single-instance stuff.
Five years later, most DBAs needed to know RAC/ASM/Data Gurd if they wanted to be Production DBAs - they then had to understand single-instance, cluster, ASM and Data Guard stuff.
I can see that within a couple of years, DBAs will have to be competent in Exadata or similar - so they'll need to understand single-instance, cluster, ASM, Data Guard AND Exadata stuff (networking, storage cells, etc).
My point is that the DBA role changes as fast as any tech specialty. Thankfully, Oracle's is making things easier to administer as the technologies we have to master become more complex: I can vouch for how difficult 9.2.0.1 was to set up for RAC - managing an 11.2 system is SO much nicer!
Unless you're working in a place that have no plans to upgrade their databases (due to legacy systems) or to grow as a business, then the chances are you will eventually be exposed to what you're looking for. Of course, if you're working in such a place, then it might be suggested that you're better off finding somewhere else to work. To stand still is to roll backwards, etc....
Good luck. It's good to be ambitious, but don't forget that a DBA role is one that absolutely demands experience - if you've 3 years of being a Production DBA and having to get THAT call in the middle of the night, you've probably got better experience than someone who has spent 6 years as a Development DBA (IMO).
Mark

Rebuild Oracle 11g R1 clustered DB

I have oracle 11g R1 clustered(two nodes) on Linux boxes, with ASM. Now the cluster is down due to lost connection to SAN. We might not be able to restore the SAN the way as it was, meaning we lost all datafiles(diskgroup). I can not start cluster, asm and DB.
We have backup(rman and export) from production. I'm trying to find the best way to rebuild this clustered database.
Thank you for your help!
Hanping

I have oracle 11g R1 clustered(two nodes) on Linux boxes, with ASM. Now the cluster is down due to lost connection to SAN. We might not be able to restore the SAN the way as it was, meaning we lost all datafiles(diskgroup). I can not start cluster, asm and DB.Where the backup is exist? is it also in disk groups or local file system?
We have backup(rman and export) from production. I'm trying to find the best way to rebuild this clustered database.After setting up everything like, creating disk groups, You can go with two options. However you have backup of RMAN & EXPORT
1) export/import
You can create a RAC database using DBCA and then import the dumpfile.
2) RMAN.
You can restore the database on one node, later you can covnert this database as RAC

Cluster / ASM restore

Similar Messages

Maybe you are looking for