Data corrupt block

Question

os Sun 5.10 oracle version 10.2.0.2 RAC 2 node
alert.log 내용
Hex dump of (file 206, block 393208) in trace file /oracle/app/oracle/admin/DBPGIC/udump/dbpgic1_ora_1424.trc
Corrupt block relative dba: 0x3385fff8 (file 206, block 393208)
Bad header found during backing up datafile
Data in bad block:
type: 32 format: 0 rdba: 0x00000001
last change scn: 0x0000.98b00394 seq: 0x0 flg: 0x00
spare1: 0x1 spare2: 0x27 spare3: 0x2
consistency value in tail: 0x00000001
check value in block header: 0x0
block checksum disabled
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
corrupt 발생한 Block id 를 검색해 보면 Block id 가 검색이 안됩니다.
dba_extents 로 검색
corrupt 때문에 Block id 가 검색이 안되는 것인지 궁금합니다.
export 받으면 데이타는 정상적으로 export 가능.

Answer 1

다행이네요. block corruption 이 발생한 곳이 데이터가 저장된 블록이
아닌 것 같습니다. 그것도 rman백업을 통해서 발견한 것 같는데
맞는지요?
scn이 scn: 0x0000.00000000 가 아닌
0x0000.98b00394 인 것으로 봐서는 physical corrupt 가 아닌
soft corrupt인 것 같습니다.
그렇다면 버그일 가능성이 높아서 찾아보니
Bug 4411228 - Block corruption with mixture of file system and RAW files
의 버그가 발견되었습니다. 이것이 아닐 수도 있지만..
이러한 block corruption에 대한 처리방법 및 원인분석은
오라클(주)를 통해서 정식으로 요청하셔야 합니다.
metalink를 통해서 SR 요청을 하십시오.
export는 high water mark 이후의 block corruption을 찾아내지 못하고 이외에도
아래 몇가지 경우에서도 찾아내지 못합니다.
db verify( dbv)의 경우에는 physical corruption은 찾아내지 못하고
soft block corruption만 찾아낼 수 있습니다.
경험상 physical corruption 이 발생하였으나 /dev/null로
datafile copy가 안되는데도 dbv로는 이 문제를 찾아내지
못하였습니다.
그렇다면 가장 좋은 방법은 rman 입니다. rman은 high water mark까지의
데이터를 백업해주면서 전체 데이터파일에 대한 체크를 하기도 합니다.
physical corruption뿐만 아니라 logical corruption도 체크를
하니 점검하기로는 rman이 가장 좋은 방법이라 생각합니다.
The Export Utility
# Use a full export to check database consistency
# Export performs a full scan for all tables
# Export only reads:
- User data below the high-water mark
- Parts of the data dictionary, while looking up information concerning the objects being exported
# Export does not detect the following:
- Disk corruptions above the high-water mark
- Index corruptions
- Free or temporary extent corruptions
- Column data corruption (like invalid date values)
block corruption을 정상적으로 복구하는 방법은 restore 후에
복구하는 방법이 있겠으나 이미 restore할 백업이 block corruption이
발생했을 수도 있습니다. 그러므로 다른 서버에 restore해보고
정상적인 datafile인 것을 확인 후에 실환경에 restore하는 것이 좋습니다.
만약 백업본까지 block corruption이 발생하였거나 또는 시간적 여유가
없을 경우에는 table을 move tablespace 또는 index rebuild를 통해서
다른 테이블스페이스로 데이터를 옮겨두고 문제가 발생한 테이블스페이스를
drop해버리고 재생성 하는 것이 좋을 것 같습니다.(지금 현재 데이터의
손실은 없으니 move tablespace, rebuild index 방법이 좋겠습니다.
Handling Corruptions
Check the alert file and system log file
Use diagnostic tools to determine the type of corruption
Dump blocks to find out what is wrong
Determine whether the error persists by running checks multiple times
Recover data from the corrupted object if necessary
Preferred resolution method: media recovery
Handling Corruptions
Always try to find out if the error is permanent. Run the analyze command multiple times or, if possible, perform a shutdown and a startup and try again to perform the operation that failed earlier.
Find out whether there are more corruptions. If you encounter one, there may be other corrupted blocks, as well. Use tools like DBVERIFY for this.
Before you try to salvage the data, perform a block dump as evidence to identify the actual cause of the corruption.
Make a hex dump of the bad block, using UNIX dd and od -x.
Consider performing a redo log dump to check all the changes that were made to the block so that you can discover when the corruption occurred.
Note: Remember that when you have a block corruption, performing media recovery is the recommended process after the hardware is verified.
Resolve any hardware issues:
- Memory boards
- Disk controllers
- Disks
Recover or restore data from the corrupt object if necessary
Handling Corruptions (continued)
There is no point in continuing to work if there are hardware failures. When you encounter hardware problems, the vendor should be contacted and the machine should be checked and fixed before continuing. A full hardware diagnostics should be run.
Many types of hardware failures are possible:
Bad I/O hardware or firmware
Operating system I/O or caching problem
Memory or paging problems
Disk repair utilities
아래 관련 자료를 드립니다.
All About Data Blocks Corruption in Oracle
Vijaya R. Dumpa
Data Block Overview:
Oracle allocates logical database space for all data in a database. The units of database space allocation are data blocks (also called logical blocks, Oracle blocks, or pages), extents, and segments. The next level of logical database space is an extent. An extent is a specific number of contiguous data blocks allocated for storing a specific type of information. The level of logical database storage above an extent is called a segment. The high water mark is the boundary between used and unused space in a segment.
The header contains general block information, such as the block address and the type of segment (for example, data, index, or rollback).
Table Directory, this portion of the data block contains information about the table having rows in this block.
Row Directory, this portion of the data block contains information about the actual rows in the block (including addresses for each row piece in the row data area).
Free space is allocated for insertion of new rows and for updates to rows that require additional space.
Row data, this portion of the data block contains rows in this block.
Analyze the Table structure to identify block corruption:
By analyzing the table structure and its associated objects, you can perform a detailed check of data blocks to identify block corruption:
SQL> analyze table_name/index_name/cluster_name ... validate structure cascade;
Detecting data block corruption using the DBVERIFY Utility:
DBVERIFY is an external command-line utility that performs a physical data structure integrity check on an offline database. It can be used against backup files and online files. Integrity checks are significantly faster if you run against an offline database.
Restrictions:
DBVERIFY checks are limited to cache-managed blocks. It’s only for use with datafiles, it will not work against control files or redo logs.
The following example is sample output of verification for the data file system_ts_01.dbf. And its Start block is 9 and end block is 25. Blocksize parameter is required only if the file to be verified has a non-2kb block size. Logfile parameter specifies the file to which logging information should be written. The feedback parameter has been given the value 2 to display one dot on the screen for every 2 blocks processed.
$ dbv file=system_ts_01.dbf start=9 end=25 blocksize=16384 logfile=dbvsys_ts.log feedback=2
DBVERIFY: Release 8.1.7.3.0 - Production on Fri Sep 13 14:11:52 2002
(c) Copyright 2000 Oracle Corporation. All rights reserved.
Output:
$ pg dbvsys_ts.log
DBVERIFY: Release 8.1.7.3.0 - Production on Fri Sep 13 14:11:52 2002
(c) Copyright 2000 Oracle Corporation. All rights reserved.
DBVERIFY - Verification starting : FILE = system_ts_01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 17
Total Pages Processed (Data) : 10
Total Pages Failing (Data) : 0
Total Pages Processed (Index) : 2
Total Pages Failing (Index) : 0
Total Pages Processed (Other) : 5
Total Pages Empty : 0
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Detecting and reporting data block corruption using the DBMS_REPAIR package:
Note: Note that this event can only be used if the block "wrapper" is marked corrupt.
Eg: If the block reports ORA-1578.
1. Create DBMS_REPAIR administration tables:
To Create Repair tables, run the below package.
SQL> EXEC DBMS_REPAIR.ADMIN_TABLES(‘REPAIR_ADMIN’, 1,1, ‘REPAIR_TS’);
Note that table names prefix with ‘REPAIR_’ or ‘ORPAN_’. If the second variable is 1, it will create ‘REAIR_key tables, if it is 2, then it will create ‘ORPAN_key tables.
If the thread variable is
1 then package performs ‘create’ operations.
2 then package performs ‘delete’ operations.
3 then package performs ‘drop’ operations.
2. Scanning a specific table or Index using the DBMS_REPAIR.CHECK_OBJECT procedure:
In the following example we check the table employee for possible corruption’s that belongs to the schema TEST. Let’s assume that we have created our administration tables called REPAIR_ADMIN in schema SYS.
To check the table block corruption use the following procedure:
SQL> VARIABLE A NUMBER;
SQL> EXEC DBMS_REPAIR.CHECK_OBJECT (‘TEST’,’EMP’, NULL,
1,’REPAIR_ADMIN’, NULL, NULL, NULL, NULL,:A);
SQL> PRINT A;
To check which block is corrupted, check in the REPAIR_ADMIN table.
SQL> SELECT * FROM REPAIR_ADMIN;
3. Fixing corrupt block using the DBMS_REPAIR.FIX_CORRUPT_BLOCK procedure:
SQL> VARIABLE A NUMBER;
SQL> EXEC DBMS_REPAIR.FIX.CORRUPT_BLOCKS (‘TEST’,’EMP’, NULL,
1,’REPARI_ADMIN’, NULL,:A);
SQL> SELECT MARKED FROM REPAIR_ADMIN;
If u select the EMP table now you still get the error ORA-1578.
4. Skipping corrupt blocks using the DBMS_REPAIR. SKIP_CORRUPT_BLOCK procedure:
SQL> EXEC DBMS_REPAIR. SKIP_CORRUPT.BLOCKS (‘TEST’, ‘EMP’, 1,1);
Notice the verification of running the DBMS_REPAIR tool. You have lost some of data. One main advantage of this tool is that you can retrieve the data past the corrupted block. However we have lost some data in the table.
5. This procedure is useful in identifying orphan keys in indexes that are pointing to corrupt rows of the table:
SQL> EXEC DBMS_REPAIR. DUMP ORPHAN_KEYS (‘TEST’,’IDX_EMP’, NULL,
2, ‘REPAIR_ADMIN’, ‘ORPHAN_ADMIN’, NULL,:A);
If u see any records in ORPHAN_ADMIN table you have to drop and re-create the index to avoid any inconsistencies in your queries.
6. The last thing you need to do while using the DBMS_REPAIR package is to run the DBMS_REPAIR.REBUILD_FREELISTS procedure to reinitialize the free list details in the data dictionary views.
SQL> EXEC DBMS_REPAIR.REBUILD_FREELISTS (‘TEST’,’EMP’, NULL, 1);
NOTE
Setting events 10210, 10211, 10212, and 10225 can be done by adding the following line for each event in the init.ora file:
Event = "event_number trace name errorstack forever, level 10"
When event 10210 is set, the data blocks are checked for corruption by checking their integrity. Data blocks that don't match the format are marked as soft corrupt.
When event 10211 is set, the index blocks are checked for corruption by checking their integrity. Index blocks that don't match the format are marked as soft corrupt.
When event 10212 is set, the cluster blocks are checked for corruption by checking their integrity. Cluster blocks that don't match the format are marked as soft corrupt.
When event 10225 is set, the fet$ and uset$ dictionary tables are checked for corruption by checking their integrity. Blocks that don't match the format are marked as soft corrupt.
Set event 10231 in the init.ora file to cause Oracle to skip software- and media-corrupted blocks when performing full table scans:
Event="10231 trace name context forever, level 10"
Set event 10233 in the init.ora file to cause Oracle to skip software- and media-corrupted blocks when performing index range scans:
Event="10233 trace name context forever, level 10"
To dump the Oracle block you can use below command from 8.x on words:
SQL> ALTER SYSTEM DUMP DATAFILE 11 block 9;
This command dumps datablock 9 in datafile11, into USER_DUMP_DEST directory.
Dumping Redo Logs file blocks:
SQL> ALTER SYSTEM DUMP LOGFILE ‘/usr/oracle8/product/admin/udump/rl. log’;
Rollback segments block corruption, it will cause problems (ORA-1578) while starting up the database.
With support of oracle, can use below under source parameter to startup the database.
CORRUPTEDROLLBACK_SEGMENTS=(RBS_1, RBS_2)
DB_BLOCK_COMPUTE_CHECKSUM
This parameter is normally used to debug corruption’s that happen on disk.
The following V$ views contain information about blocks marked logically corrupt:
V$ BACKUP_CORRUPTION, V$COPY_CORRUPTION
When this parameter is set, while reading a block from disk to catch, oracle will compute the checksum again and compares it with the value that is in the block.
If they differ, it indicates that the block is corrupted on disk. Oracle makes the block as corrupt and signals an error. There is an overhead involved in setting this parameter.
DB_BLOCK_CACHE_PROTECT=‘TRUE’
Oracle will catch stray writes made by processes in the buffer catch.
Oracle 9i new RMAN futures:
Obtain the datafile numbers and block numbers for the corrupted blocks. Typically, you obtain this output from the standard output, the alert.log, trace files, or a media management interface. For example, you may see the following in a trace file:
ORA-01578: ORACLE data block corrupted (file # 9, block # 13)
ORA-01110: data file 9: '/oracle/dbs/tbs_91.f'
ORA-01578: ORACLE data block corrupted (file # 2, block # 19)
ORA-01110: data file 2: '/oracle/dbs/tbs_21.f'
$rman target =rman/rman@rmanprod
RMAN> run {
2> allocate channel ch1 type disk;
3> blockrecover datafile 9 block 13 datafile 2 block 19;
4> }
Recovering Data blocks Using Selected Backups:
# restore from backupset
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 FROM BACKUPSET;
# restore from datafile image copy
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 FROM DATAFILECOPY;
# restore from backupset with tag "mondayAM"
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 199 FROM TAG = mondayAM;
# restore using backups made before one week ago
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE
UNTIL 'SYSDATE-7';
# restore using backups made before SCN 100
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE UNTIL SCN 100;
# restore using backups made before log sequence 7024
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE
UNTIL SEQUENCE 7024;
글 수정:
Min Angel (Yeon Hong Min, Korean)