Corrupt row

Hello I have the following table that has over 20 million rows in it. one of the rows is corrupt. How can I remove this corrupt row. Copying the data to a new table and excluding this rowid just takes too lond and I eventually end up getting a Snapshot too old error.
Here is the table def:
Name                                      Null?    Type
RETAILER_ID                               NOT NULL NUMBER
SKU_ID                                    NOT NULL VARCHAR2(18)
OUTLET_ID                                 NOT NULL VARCHAR2(20)
QTY_CHANGE                                NOT NULL NUMBER
TRANS_DATE                                         DATEI know that the exact corrupt row id is - 'AAALB0AASAAAIepAAA'
I also tried to use an export with a where clause to exclude this row, but it also failed. Any ideas/tricks would be greatly appreciated. Thank you.
David

Thanks, just making sure.
I like the rman recommendation from the other poster.
How did you come to find out the row is corrupt? Are you sure it is not an index corruption?
How are you selecting/deleting the row? By rowid?
How are you trying to copy the data into the other table?

Similar Messages

Identifying corrupted rows and retrieving the remainder

Hello all.
Please give some help.
The alert log shows that
a datafile has a block corrupted which belongs to one of the three primary tables used for reporting and which for some reason hasnt got a backup.
I need to find out which rows
are involved in the corruption so that to create a table from the corrupted with the condition like
AND ROWID NOT IN (list of corrupted ROWIDs)Please reply as sson as possible.
Thanks much.

Hello again.
Unfortunately I wasnt able to see the text of your last messages, seems smth is wrong with my browser, although I can open other pages....anyway just wanted so say that I used
DBMS_REPAIR SKIP SCAN to dicard the inaccessible rows then I created a copy of the table then droped the original one, created it again and inserted data back into it. After that the alert log stopped writing about the corrupted block.
But one thing still remains unclear to me. Where did that block gone? Did oracle deallocate it after I dropped the table or is there any other reason for it?
PLease give an axplanation.
Thanks much.

How to impdp a specific table excluding rows which are corrupted using impdp

Hello,
I need some help with regard to the impdp. I have table which is corrupted and the information of corrupted rows is also available with me now i need to exclude those rows in specific and import the other error free rows.
Can you please let me know what would be the option to use with impdp.
Does the QUERY option work with impdp to exclude rows of that particular table.
Thanks,
Vinodh

but import is running for more than one day but the size of table is huge around 600GB.
Below you can find the parfile output. How to check if this process is still running. I have used dba_datapump_jobs but no use.
also truss output says thread is sleeping. Your inputs on this will be much helpful.
DIRECTORY=DUMPDIR113
DUMPFILE=expdpA_DDAPTE1_20130403_1003_%U.dmp
TABLES=A_DDAPTE1.FGBA_JBBB069
TABLE_EXISTS_ACTION=APPEND
QUERY=A_DDAPTE1.FGBA_JBBB069:"WHERE RECID NOT IN ('600KR7Feb07-5010009050','FTS100913QA-1','FTS100913QA-11','FTS100913QA-14','FTS100913QA-15','FTS100913QA-16'
,'FTS100913QA-17','FTS100913QA-19','FTS100913QA-2','FTS100913QA-25','FTS100913QA-3','FTS100913QA-31','LnLmtTst4Mar06-LMT-1015000','LndPrdMixDec16-4510003850',
'LndPrdMixDec16-4510004200','LndPrdMixDec30-4400041826','LndPrdMixJan08-1080005191','LndPrdMixJan19-1020043784','LndPrdMixJan24-1300009271','LndPrdMixJan27-20
80004131','LndPrdMixFeb08-2200008537','LndPrdMixFeb12-2061035772','LndPrdMixFeb23-4010020173','LndPrdMixFeb27-4030043858','LnLmtTst4Mar06-LMT-1015000','LndPdM
xMr17-1-2040044689','JPMF12309051824404','LndPrdMixMar31-4100030945','LndPrdMixApr26-5020016018','LndPdMxJl09-1-2090030535','JPMJ103090619303826','LndPrdMixSe
p03-4070048268','LndPrdMixSep24-4400010477','FTS100913QA-14','FTS100913QA-16','FTS100913QA-2','FTS100913QA-17','FTS100913QA-15','FTS100913QA-25','FTS100913QA-
31','FTS100913QA-11','FTS100913QA-19','FTS100913QA-3','FTS100913QA-1','JPMI22909071376392','LndPrdMixDec16-4510003850','LndPrdMixDec16-4510004200','LndPrdMixD
ec22-5020009996','LndPdMxJ317-1-2090007173','LndYr2011Jan18-4070033301','LndYr2011Feb02-4400000479','Lndn-11-671BKFeb24-2-1000091345','LndYr2011Mar11-41000182
11','RPS09090401918928','LndYr2011Feb26-3050008971','LndYr2011Mar214-4400016890','LndYr2011Apr08-4100023458','FTS09091801399914','LndYr2011May26-5010040755','
LndYr2011Jun01-4030034285','Thrd25BulkJun30-5010048461','CobRun0211Jul08-5020030807','600KCsp511Aug09-4030048518','RlTime250k2Sep09-6010025105','CCS0911060114
460','3600KR61284NOV24-4030000342','600KR7Feb07-5010009050','FTS09120801346378','600KFWDDATEDMAR09-5010016394','ST10912240129034','MMK0912300369630','600KR701
MAY04-4100000188','GBA671Tes2RemDor-3000088468','R21210JUL2012R1-3200014672','CBG1002030116505','R312G10OCT2012R3-3070047935','R312G30OCT2012RR1-3090016437','
SOLARIS07DEC2012RR1-3200011526','LINUX23JAN2013RR1-3020015180','LINUX29JAN2013RR1-3050004304','LINUX04FEB2013RT2-3020045579','MMK10032201164879','FTS100324012
16453','MMB1003260224532','LINUX14FEB2013RB2-3060003239','LINUX25FEB2013RB1-3100048142','FTS10040802260630','LINUX24MAR2013RB1-4400044062','LINUX23MAY2013RR1-
3060001245','LINUX10JUN2013RR3-3030035797','LINUX12JUN2013RR2-3060024808','LINUX14JUN2013RR2-3040046259')"
LOGFILE=YYYY069_20130702LOG.log
PARALLEL=15
Thanks,
Vinodh

How to import a corrupted file?

I got the following errors when I did import.
It seems that the file was corrupted.
But do you know if there are some tools or methods which can skip the corrupted rows and do the rest of the file.
Thanks,
Samuel
Connected to: Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
With the Partitioning option
JServer Release 9.2.0.4.0 - Production
Export file created by EXPORT:V08.01.07 via direct path
import done in ZHT16BIG5 character set and AL16UTF16 NCHAR character set
import server uses WE8ISO8859P1 character set (possible charset conversion)
export server uses ZHT16BIG5 NCHAR character set (possible ncharset conversion)
. . importing table "RTX_020401"
IMP-00009: abnormal end of export file
IMP-00018: partial import of previous table completed: 7340500 rows imported
Import terminated successfully with warnings.

The dump file are read it sequentially and you can not skip to certain level. You could try to import some kind of objects and whether those objects are not in the sector damaged of the file , they will be able to be loaded.
Joel Pérez

Screen corrupt 2010 Mac Mini HDMI (Static/Snow/White Noise)

I've just upgraded my 2010 mac mini to Lion, this is connected to a projector via HDMI to a Denon amp. Before the upgrade there were no issues the picture was completely clear and was usable in all resolutions.
Now that lion has installed the display is not usable, all I can see is 99% static and a corrupt row at the top of the screen where the menu bar can be just about made out.
Nothing has changed to the setup other than installing Lion, if I use screen sharing I can see the desktop happily, the OS is fully responsive.
I haven't yet tried to connect it to another monitor as I don't have a spare, I'll attempt this over the weekend, has anyone else experienced this issue?

I've got a similar issue and seems like a few others also have problems as well
https://discussions.apple.com/thread/3196192?tstart=0
My Mac Mini wont boot to HDMI and Screen Sharing wont link in since Lion upgrade......
A makeshift solution is unplugging the HDMI during boot up, give it 30sec's - 1min...plug it back it and then it should come up.... works for me for now.....
hopefully someone has a more permanent solution or mac solves the issue ASAP....
Cheers

Data corrupt block

os Sun 5.10 oracle version 10.2.0.2 RAC 2 node
alert.log 내용
Hex dump of (file 206, block 393208) in trace file /oracle/app/oracle/admin/DBPGIC/udump/dbpgic1_ora_1424.trc
Corrupt block relative dba: 0x3385fff8 (file 206, block 393208)
Bad header found during backing up datafile
Data in bad block:
type: 32 format: 0 rdba: 0x00000001
last change scn: 0x0000.98b00394 seq: 0x0 flg: 0x00
spare1: 0x1 spare2: 0x27 spare3: 0x2
consistency value in tail: 0x00000001
check value in block header: 0x0
block checksum disabled
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
Reread of blocknum=393208, file=/dev/md/vg_rac06/rdsk/d119. found same corrupt data
corrupt 발생한 Block id 를 검색해 보면 Block id 가 검색이 안됩니다.
dba_extents 로 검색
corrupt 때문에 Block id 가 검색이 안되는 것인지 궁금합니다.
export 받으면 데이타는 정상적으로 export 가능.

다행이네요. block corruption 이 발생한 곳이 데이터가 저장된 블록이
아닌 것 같습니다. 그것도 rman백업을 통해서 발견한 것 같는데
맞는지요?
scn이 scn: 0x0000.00000000 가 아닌
0x0000.98b00394 인 것으로 봐서는 physical corrupt 가 아닌
soft corrupt인 것 같습니다.
그렇다면 버그일 가능성이 높아서 찾아보니
Bug 4411228 - Block corruption with mixture of file system and RAW files
의 버그가 발견되었습니다. 이것이 아닐 수도 있지만..
이러한 block corruption에 대한 처리방법 및 원인분석은
오라클(주)를 통해서 정식으로 요청하셔야 합니다.
metalink를 통해서 SR 요청을 하십시오.
export는 high water mark 이후의 block corruption을 찾아내지 못하고 이외에도
아래 몇가지 경우에서도 찾아내지 못합니다.
db verify( dbv)의 경우에는 physical corruption은 찾아내지 못하고
soft block corruption만 찾아낼 수 있습니다.
경험상 physical corruption 이 발생하였으나 /dev/null로
datafile copy가 안되는데도 dbv로는 이 문제를 찾아내지
못하였습니다.
그렇다면 가장 좋은 방법은 rman 입니다. rman은 high water mark까지의
데이터를 백업해주면서 전체 데이터파일에 대한 체크를 하기도 합니다.
physical corruption뿐만 아니라 logical corruption도 체크를
하니 점검하기로는 rman이 가장 좋은 방법이라 생각합니다.
The Export Utility
# Use a full export to check database consistency
# Export performs a full scan for all tables
# Export only reads:
- User data below the high-water mark
- Parts of the data dictionary, while looking up information concerning the objects being exported
# Export does not detect the following:
- Disk corruptions above the high-water mark
- Index corruptions
- Free or temporary extent corruptions
- Column data corruption (like invalid date values)
block corruption을 정상적으로 복구하는 방법은 restore 후에
복구하는 방법이 있겠으나 이미 restore할 백업이 block corruption이
발생했을 수도 있습니다. 그러므로 다른 서버에 restore해보고
정상적인 datafile인 것을 확인 후에 실환경에 restore하는 것이 좋습니다.
만약 백업본까지 block corruption이 발생하였거나 또는 시간적 여유가
없을 경우에는 table을 move tablespace 또는 index rebuild를 통해서
다른 테이블스페이스로 데이터를 옮겨두고 문제가 발생한 테이블스페이스를
drop해버리고 재생성 하는 것이 좋을 것 같습니다.(지금 현재 데이터의
손실은 없으니 move tablespace, rebuild index 방법이 좋겠습니다.
Handling Corruptions
Check the alert file and system log file
Use diagnostic tools to determine the type of corruption
Dump blocks to find out what is wrong
Determine whether the error persists by running checks multiple times
Recover data from the corrupted object if necessary
Preferred resolution method: media recovery
Handling Corruptions
Always try to find out if the error is permanent. Run the analyze command multiple times or, if possible, perform a shutdown and a startup and try again to perform the operation that failed earlier.
Find out whether there are more corruptions. If you encounter one, there may be other corrupted blocks, as well. Use tools like DBVERIFY for this.
Before you try to salvage the data, perform a block dump as evidence to identify the actual cause of the corruption.
Make a hex dump of the bad block, using UNIX dd and od -x.
Consider performing a redo log dump to check all the changes that were made to the block so that you can discover when the corruption occurred.
Note: Remember that when you have a block corruption, performing media recovery is the recommended process after the hardware is verified.
Resolve any hardware issues:
- Memory boards
- Disk controllers
- Disks
Recover or restore data from the corrupt object if necessary
Handling Corruptions (continued)
There is no point in continuing to work if there are hardware failures. When you encounter hardware problems, the vendor should be contacted and the machine should be checked and fixed before continuing. A full hardware diagnostics should be run.
Many types of hardware failures are possible:
Bad I/O hardware or firmware
Operating system I/O or caching problem
Memory or paging problems
Disk repair utilities
아래 관련 자료를 드립니다.
All About Data Blocks Corruption in Oracle
Vijaya R. Dumpa
Data Block Overview:
Oracle allocates logical database space for all data in a database. The units of database space allocation are data blocks (also called logical blocks, Oracle blocks, or pages), extents, and segments. The next level of logical database space is an extent. An extent is a specific number of contiguous data blocks allocated for storing a specific type of information. The level of logical database storage above an extent is called a segment. The high water mark is the boundary between used and unused space in a segment.
The header contains general block information, such as the block address and the type of segment (for example, data, index, or rollback).
Table Directory, this portion of the data block contains information about the table having rows in this block.
Row Directory, this portion of the data block contains information about the actual rows in the block (including addresses for each row piece in the row data area).
Free space is allocated for insertion of new rows and for updates to rows that require additional space.
Row data, this portion of the data block contains rows in this block.
Analyze the Table structure to identify block corruption:
By analyzing the table structure and its associated objects, you can perform a detailed check of data blocks to identify block corruption:
SQL> analyze table_name/index_name/cluster_name ... validate structure cascade;
Detecting data block corruption using the DBVERIFY Utility:
DBVERIFY is an external command-line utility that performs a physical data structure integrity check on an offline database. It can be used against backup files and online files. Integrity checks are significantly faster if you run against an offline database.
Restrictions:
DBVERIFY checks are limited to cache-managed blocks. It’s only for use with datafiles, it will not work against control files or redo logs.
The following example is sample output of verification for the data file system_ts_01.dbf. And its Start block is 9 and end block is 25. Blocksize parameter is required only if the file to be verified has a non-2kb block size. Logfile parameter specifies the file to which logging information should be written. The feedback parameter has been given the value 2 to display one dot on the screen for every 2 blocks processed.
$ dbv file=system_ts_01.dbf start=9 end=25 blocksize=16384 logfile=dbvsys_ts.log feedback=2
DBVERIFY: Release 8.1.7.3.0 - Production on Fri Sep 13 14:11:52 2002
(c) Copyright 2000 Oracle Corporation. All rights reserved.
Output:
$ pg dbvsys_ts.log
DBVERIFY: Release 8.1.7.3.0 - Production on Fri Sep 13 14:11:52 2002
(c) Copyright 2000 Oracle Corporation. All rights reserved.
DBVERIFY - Verification starting : FILE = system_ts_01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 17
Total Pages Processed (Data) : 10
Total Pages Failing (Data) : 0
Total Pages Processed (Index) : 2
Total Pages Failing (Index) : 0
Total Pages Processed (Other) : 5
Total Pages Empty : 0
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Detecting and reporting data block corruption using the DBMS_REPAIR package:
Note: Note that this event can only be used if the block "wrapper" is marked corrupt.
Eg: If the block reports ORA-1578.
1. Create DBMS_REPAIR administration tables:
To Create Repair tables, run the below package.
SQL> EXEC DBMS_REPAIR.ADMIN_TABLES(‘REPAIR_ADMIN’, 1,1, ‘REPAIR_TS’);
Note that table names prefix with ‘REPAIR_’ or ‘ORPAN_’. If the second variable is 1, it will create ‘REAIR_key tables, if it is 2, then it will create ‘ORPAN_key tables.
If the thread variable is
1 then package performs ‘create’ operations.
2 then package performs ‘delete’ operations.
3 then package performs ‘drop’ operations.
2. Scanning a specific table or Index using the DBMS_REPAIR.CHECK_OBJECT procedure:
In the following example we check the table employee for possible corruption’s that belongs to the schema TEST. Let’s assume that we have created our administration tables called REPAIR_ADMIN in schema SYS.
To check the table block corruption use the following procedure:
SQL> VARIABLE A NUMBER;
SQL> EXEC DBMS_REPAIR.CHECK_OBJECT (‘TEST’,’EMP’, NULL,
1,’REPAIR_ADMIN’, NULL, NULL, NULL, NULL,:A);
SQL> PRINT A;
To check which block is corrupted, check in the REPAIR_ADMIN table.
SQL> SELECT * FROM REPAIR_ADMIN;
3. Fixing corrupt block using the DBMS_REPAIR.FIX_CORRUPT_BLOCK procedure:
SQL> VARIABLE A NUMBER;
SQL> EXEC DBMS_REPAIR.FIX.CORRUPT_BLOCKS (‘TEST’,’EMP’, NULL,
1,’REPARI_ADMIN’, NULL,:A);
SQL> SELECT MARKED FROM REPAIR_ADMIN;
If u select the EMP table now you still get the error ORA-1578.
4. Skipping corrupt blocks using the DBMS_REPAIR. SKIP_CORRUPT_BLOCK procedure:
SQL> EXEC DBMS_REPAIR. SKIP_CORRUPT.BLOCKS (‘TEST’, ‘EMP’, 1,1);
Notice the verification of running the DBMS_REPAIR tool. You have lost some of data. One main advantage of this tool is that you can retrieve the data past the corrupted block. However we have lost some data in the table.
5. This procedure is useful in identifying orphan keys in indexes that are pointing to corrupt rows of the table:
SQL> EXEC DBMS_REPAIR. DUMP ORPHAN_KEYS (‘TEST’,’IDX_EMP’, NULL,
2, ‘REPAIR_ADMIN’, ‘ORPHAN_ADMIN’, NULL,:A);
If u see any records in ORPHAN_ADMIN table you have to drop and re-create the index to avoid any inconsistencies in your queries.
6. The last thing you need to do while using the DBMS_REPAIR package is to run the DBMS_REPAIR.REBUILD_FREELISTS procedure to reinitialize the free list details in the data dictionary views.
SQL> EXEC DBMS_REPAIR.REBUILD_FREELISTS (‘TEST’,’EMP’, NULL, 1);
NOTE
Setting events 10210, 10211, 10212, and 10225 can be done by adding the following line for each event in the init.ora file:
Event = "event_number trace name errorstack forever, level 10"
When event 10210 is set, the data blocks are checked for corruption by checking their integrity. Data blocks that don't match the format are marked as soft corrupt.
When event 10211 is set, the index blocks are checked for corruption by checking their integrity. Index blocks that don't match the format are marked as soft corrupt.
When event 10212 is set, the cluster blocks are checked for corruption by checking their integrity. Cluster blocks that don't match the format are marked as soft corrupt.
When event 10225 is set, the fet$ and uset$ dictionary tables are checked for corruption by checking their integrity. Blocks that don't match the format are marked as soft corrupt.
Set event 10231 in the init.ora file to cause Oracle to skip software- and media-corrupted blocks when performing full table scans:
Event="10231 trace name context forever, level 10"
Set event 10233 in the init.ora file to cause Oracle to skip software- and media-corrupted blocks when performing index range scans:
Event="10233 trace name context forever, level 10"
To dump the Oracle block you can use below command from 8.x on words:
SQL> ALTER SYSTEM DUMP DATAFILE 11 block 9;
This command dumps datablock 9 in datafile11, into USER_DUMP_DEST directory.
Dumping Redo Logs file blocks:
SQL> ALTER SYSTEM DUMP LOGFILE ‘/usr/oracle8/product/admin/udump/rl. log’;
Rollback segments block corruption, it will cause problems (ORA-1578) while starting up the database.
With support of oracle, can use below under source parameter to startup the database.
CORRUPTEDROLLBACK_SEGMENTS=(RBS_1, RBS_2)
DB_BLOCK_COMPUTE_CHECKSUM
This parameter is normally used to debug corruption’s that happen on disk.
The following V$ views contain information about blocks marked logically corrupt:
V$ BACKUP_CORRUPTION, V$COPY_CORRUPTION
When this parameter is set, while reading a block from disk to catch, oracle will compute the checksum again and compares it with the value that is in the block.
If they differ, it indicates that the block is corrupted on disk. Oracle makes the block as corrupt and signals an error. There is an overhead involved in setting this parameter.
DB_BLOCK_CACHE_PROTECT=‘TRUE’
Oracle will catch stray writes made by processes in the buffer catch.
Oracle 9i new RMAN futures:
Obtain the datafile numbers and block numbers for the corrupted blocks. Typically, you obtain this output from the standard output, the alert.log, trace files, or a media management interface. For example, you may see the following in a trace file:
ORA-01578: ORACLE data block corrupted (file # 9, block # 13)
ORA-01110: data file 9: '/oracle/dbs/tbs_91.f'
ORA-01578: ORACLE data block corrupted (file # 2, block # 19)
ORA-01110: data file 2: '/oracle/dbs/tbs_21.f'
$rman target =rman/rman@rmanprod
RMAN> run {
2> allocate channel ch1 type disk;
3> blockrecover datafile 9 block 13 datafile 2 block 19;
4> }
Recovering Data blocks Using Selected Backups:
# restore from backupset
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 FROM BACKUPSET;
# restore from datafile image copy
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 FROM DATAFILECOPY;
# restore from backupset with tag "mondayAM"
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 199 FROM TAG = mondayAM;
# restore using backups made before one week ago
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE
UNTIL 'SYSDATE-7';
# restore using backups made before SCN 100
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE UNTIL SCN 100;
# restore using backups made before log sequence 7024
BLOCKRECOVER DATAFILE 9 BLOCK 13 DATAFILE 2 BLOCK 19 RESTORE
UNTIL SEQUENCE 7024;
글 수정:
Min Angel (Yeon Hong Min, Korean)

SQL server 2014 and VS 2013 - Dataflow task, read CSV file and insert data to SQL table

Hello everyone,
I was assigned a work item wherein, I've a dataflow task on For Each Loop container at control flow of SSIS package. This For Each Loop container reads the CSV files from the specified location one by one, and populates a variable with current
file name. Note, the tables where I would like to push the data from each CSV file are also having the same names as CSV file names.
On the dataflow task, I've Flat File component as a source, this component uses the above variable to read the data of a particular file. Now, here my question comes, how can I move the data to destination, SQL table, using the same variable name?
I've tried to setup the OLE DB destination component dynamically but it executes well only for first time. It does not change the mappings as per the columns of the second CSV file. There're around 50 CSV files, each has different set off columns
in it. These files needs to be migrated to SQL tables using the optimum way.
Does anybody know which is the best way to setup the Dataflow task for this requirement?
Also, I cannot use Bulk insert task here as we would like to keep a log of corrupted rows.
Any help would be much appreciated. It's very urgent.
Thanks, <b>Ankit Shah</b> <hr> Inkey Solutions, India. <hr> Microsoft Certified Business Management Solutions Professionals <hr> http://ankit.inkeysolutions.com

The standard Data Flow Task supports only static metadata defined during design time. I would recommend you check the commercial COZYROC
Data Flow Task Plus. It is an extension of the standard Data Flow Task and it supports dynamic metadata at runtime. You can process all your input CSV files using a single Data Flow Task
Plus. No programming skills are required.
SSIS Tasks Components Scripts Services | http://www.cozyroc.com/

Read From Measurement File... removes X Values of first column?

During one of our tests, two instruments were switched at the terminal by accident. I need to read in the massive lvm files, remove the wrong scaling and apply the correct scaling, while switching the values in the columns and right it all to new files. Simple right?
I wanted to use the Read from Measurement File.vi to make things easier, because the files are very large and I would like to analyze them 100 rows at a time. Some of the the files are around 1.5 Gigs in size so I need to read them in chunks.
The read from measurement file keeps removing the first column from the data! It outputs the data as a signal (dynamic data) and I have to use the dynamic to numeric array express vi. For some reason, before I even get to that point, the first column is not in the data.
No matter what settings I pick on the read from measurement express vi, the time column is removed from the data. I have checked/unchecked the "first row is channel names" and "first column is time channel" to no avail. The odd thing is that in the preview it shows the first column, as if it will read it properly... but it doesnt. Nothing I change in the settings can seems to make a difference in getting the first column, the x values, out of the file.
Below you can see the first column completely removed from the data.
This is extremely frustrating. By probing the signal out I can see the dynamic data attributes and the time column has already been removed, so I don't think that the signal to double array express vi is the problem, but I am not sure.
I am attaching my VI and a small data file to be analyzed. You can see what I mean.
The alternatives seem less than adequate. The read from spreadsheet file vi wants an offset of a specific number of characters not rows. The problem is that this is not constant between rows for some reason, when hidden characters are taken into account so I cant just set the number of characters in 100 rows and increment the offset in a loop... like I normally would. That means I might miss data or get a corrupted row.
This means that I have to use the read from text file, read how ever many characters I think a row is (over estimating a bit) then search for the newline character, find out how many chars are in that set and then offset that for the next loop iteration, all while converting each string number to a double. Talk about slow.
I have searched around and found that I am not the only one that has had this issue. This is a common thing, but no one seems to have the answer. Why can't the read from measurement file VI read all of the numbers in every row? Why cant I tell it I want a 2d array of doubles out and not a dynamic data type? It has to be something I am doing wrong.
Attached is a zip file with my VI and two data files. The "S19_A_DSI_detensioning_c.lvm" is the one generated by my VI (_c meaning corrected). "S19_A_DSI_detensioning.lvm" is the original measurement file. I hope you will pardon my messy VI, it's a quicky.
Any help you guys can give would be much appreciated.
[will work for kudos]
Attachments:
Scaling Factor Correction.zip ‏1109 KB

That is a great workaround. The help talked about putting a check next to "read lines" but for the life of me I couldn't find where to do that. I wonder what other VIs have mystery check options in the right mouse click menu. I mean normally options like those are inputs, I thought. I'm going to start right clicking on every VI I drop to see if there are options there I never realized.
I would still have to use the set file position VI and specify the byte offset right? How would I know where that is? I guess each character is a byte and I would count the characters in the string retrieved and then offset by that amount on the next iteration using a shift register?
While waiting for help, I ended up using the read from text file and using the match string to look for the new line character, and using the spreadsheet string to array vi, analyzed the files line by line. Thats just because I couldn't easily come up with a regular expression to get 100 lines. It was slow but it worked.
However, that still really doesn't answer the question of why it is impossible to get the first column with the read from measurement file express VI. Does anyone know? Is this a known bug?
[will work for kudos]

SSIS: Why do columns become misaligned when importing flat files?

Hi All
I am stumped with the following.
When I try to load a fixed length flat file into a table, the first few thousand records load correctly but then the columns start going out of sync.
Visually it looks like the data is drifting off to the right. Below is what the table looks like when the load completes:
Col1    Col2    Col3    Col4
1        1         1         1
1        1         1         1
1        1         1         1
   1        1         1         1
    1        1         1         1
      1        1         1         1
        1        1         1         1
Additional info:
1. The file has 400 000 records.
2. The first few thousand records load OK.
3. Source file is a flat file, fixed length, no delimeters.
4. All rows are same length, with LFCR.
I tried using a script task to check with C# if all rows are the same length and has the same line terminator, and could not pick up anything out of the ordinary.
What could be the cause?

Thank you for the reply.
I discovered the source of the problem, but still don't understand how to solve it.
There is one extra character in one of the columns every few thousand lines. This is the character: �
I don't understand why all the rows following a corrupt row is shifted by one character and not just the effected row?
Secondly, via a script task, SSIS indicates that the length of the row is still the same, despite the extra character. Is it possible that SSIS does not recognize this character and this is what is causing all columns to shift / mis-align?

Modification of Array contents

I have a data array of 22 columns and typically 4000 rows. The data in the array is generated by a towed body and occasionally has corrupt rows of data. There can be several isolated groups of corrupt data. The corrupt data is easily detected as a transient in an otherwise slowly changing data stream.
The user would like to review and modify the contents of the array to either remove the corrupt rows and/or to interpolate from the last good data to the next good data,
I can identify the corrupt rows and generating the interpolating array to replace the corrupt data. I can find no way of inserting the new data back into the original array or even generating a single modified array containing all the modifications. I can generate a new array for each modification but that is not what I need. Every attempt to modify an existing array gives the error message "Member of a Circle".
Is this is due to my not understanding Labview?
I am a infrequent user of Labview.

here is my way:
the array is stored in a shift register of a while loop
one additional copy is made to cancel the changes
you can add new events (or better copy the 'replace' event) and add other functions
you don't have to use an event structure...
Greetings from Germany
Henrik
LV since v3.1
“ground” is a convenient fantasy
'˙˙˙˙uıɐƃɐ lɐıp puɐ °06 ǝuoɥd ɹnoʎ uɹnʇ ǝsɐǝld 'ʎɹɐuıƃɐɯı sı pǝlɐıp ǝʌɐɥ noʎ ɹǝqɯnu ǝɥʇ'
Attachments:
manipulate array.vi ‏49 KB

ConfigurationException

Hi,
I am getting the following error when I try to login into the portal. I
am able to see the first page on the portal. As I try to login, it throws
the following exception.
<Feb 21, 2002 5:17:36 PM IST> <Error> <Webflow> <Error while parsing uri
/powerp
athportal/application, path null, query string null - Webflow XML does not
exist
, is not loaded properly, or you do not have a configuration-error-page
defined.
Exception[com.bea.p13n.appflow.exception.ConfigurationException: The
configurati
on-error-page node was not found in the webflow xml file. for webapp [null],
nam
espace [portal]. While trying to display CONFIGURATION ERROR:
[Exception[com.bea
.p13n.appflow.exception.ConfigurationException: Bad Namespace - namespace
[porta
l] is not available for webflow execution. Make sure the [portal.wf] file is
dep
loyed in webapp [null].]],]
at
com.bea.p13n.appflow.webflow.internal.WebflowExecutorImpl.processConf
igurationError(WebflowExecutorImpl.java:772)
at
com.bea.p13n.appflow.webflow.internal.WebflowExecutorImpl.processWebf
lowRequest(WebflowExecutorImpl.java:474)
at
com.bea.portal.appflow.PortalAppflowHelper.invokeWebflow(PortalAppflo
wHelper.java:139)
at
com.bea.portal.appflow.servlets.internal.PortalWebflowServlet.doGet(P
ortalWebflowServlet.java:124)
at
com.bea.p13n.appflow.webflow.servlets.internal.WebflowServlet.doPost(
WebflowServlet.java:213)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispat
cherImpl.java:215)
at
weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:112
at jsp_servlet.__index._jspService(__index.java:116)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispat
cherImpl.java:215)
at
weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:112
at jsp_servlet.__login._jspService(__login.java:227)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppSe
rvletContext.java:2459)
at
weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestIm
pl.java:2039)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

Hello Rahul,
Did you use the EBCC to sync the portal.wf XML file to the server and the
database? If that is not the problem then, if you are using Portal 4.0 sp1 with
Oracle, then maybe you have encountered a bug involving the writing of CLOBs.
It was fixed and support has a patch, patch_CR067935.zip, that addresses it.
Here is the README from the patch:
patch_CR067935
This patch addresses an issue for the UPDATE of CLOBs when
using Oracle.
Oracle has unusual behavior for the UPDATE of CLOBs.
If you use UPDATE to put a small CLOB into a field that contains
a larger CLOB, then the CLOB size remains unchanged. In other
words, the new small CLOB data overlaps the large CLOB data and
the end of the CLOB consists of the old data from the end of the
large CLOB.
This patch modifies the OracleJdbcHelperDelegate to trim a CLOB
after updating it. The OracleJDriverJdbcHelperDelegate and
OracleThinJdbcHelperDelegate, which extend this abstract base
class, were updated to use the new trimClob() method.
Installation and Setup Procedures:
It is important that all of these steps be followed for the
service pack to be applied correctly. Failure to follow these
steps may result in unexpected behavior.
Throughout this document, <wlportal-install> refers to the
directory where wlportal 4.0 sp1 is installed on the system.
1. To apply the Service Pack on UNIX, you must have READ, WRITE,
and EXECUTE permission in the <wlportal-install> directory (and
associated subdirectories).
2. Stop the server.
3. Extract the file patch_CR067935.zip in a temp directory or
save time by extracting it directly into the <portal-install>
directory.
4. If you extracted the .zip file into a temp directory then copy
patch_CR067935.jar into <wlportal-install>/lib/
5. The replaced classes are used in your system classpath. You can
see this if you note that they originally appeared in
<wlportal-install>/lib/p13n_system.jar. This jar file is placed
in the system class path in your set-environment script.
Therefore, you can use this patch by placing it first in your system
classpath. All applications deployed on your server will load the
new classes:
Modify your set-environment script to place
<wlportal-install>/lib/patch_CR067935.jar before p13n_system.jar in
your system classpath.
For example (for Win32)...
CHANGE
REM ----------- WebLogic CLASSPATH -----------
SET
CLASSPATH=%P13N_DIR%\lib\patches.jar;%JAVA_CLASSPATH%;%EXT_CLASSPATH%;%DB_CLASSPATH%
TO
REM ----------- WebLogic CLASSPATH -----------
SET
CLASSPATH=%P13N_DIR%\lib\patch_CR067935.jar;%P13N_DIR%\lib\patches.jar;%JAVA_CLASSPATH%;%EXT_CLASSPATH%;%DB_CLASSPATH%
6. Restart your server.
7. If you have CLOBs in your DATA_SYNC_ITEM table that were corrupted by
this
bug, then you will get several DocumentProcessingException stack traces
upon
server startup. To fix this:
* note the exceptions because you can fix them by replacing the
corrupted
rows from the DATA_SYNC_ITEM table using the data sync mechanism.
The DocumentProcessingException identifies the URI of the bad XML
file.
You and can replace it in the database like this:
* Touch the file in your application-sync directory for your project to
update
the timestamp.
* Perform a data sync for this file (or files), using the EBCC.
Rahul Kapoor wrote:
Hi,
I am getting the following error when I try to login into the portal. I
am able to see the first page on the portal. As I try to login, it throws
the following exception.
<Feb 21, 2002 5:17:36 PM IST> <Error> <Webflow> <Error while parsing uri
/powerp
athportal/application, path null, query string null - Webflow XML does not
exist
, is not loaded properly, or you do not have a configuration-error-page
defined.
Exception[com.bea.p13n.appflow.exception.ConfigurationException: The
configurati
on-error-page node was not found in the webflow xml file. for webapp [null],
nam
espace [portal]. While trying to display CONFIGURATION ERROR:
[Exception[com.bea
.p13n.appflow.exception.ConfigurationException: Bad Namespace - namespace
[porta
l] is not available for webflow execution. Make sure the [portal.wf] file is
dep
loyed in webapp [null].]],]
at
com.bea.p13n.appflow.webflow.internal.WebflowExecutorImpl.processConf
igurationError(WebflowExecutorImpl.java:772)
at
com.bea.p13n.appflow.webflow.internal.WebflowExecutorImpl.processWebf
lowRequest(WebflowExecutorImpl.java:474)
at
com.bea.portal.appflow.PortalAppflowHelper.invokeWebflow(PortalAppflo
wHelper.java:139)
at
com.bea.portal.appflow.servlets.internal.PortalWebflowServlet.doGet(P
ortalWebflowServlet.java:124)
at
com.bea.p13n.appflow.webflow.servlets.internal.WebflowServlet.doPost(
WebflowServlet.java:213)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispat
cherImpl.java:215)
at
weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:112
at jsp_servlet.__index._jspService(__index.java:116)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispat
cherImpl.java:215)
at
weblogic.servlet.jsp.PageContextImpl.forward(PageContextImpl.java:112
at jsp_servlet.__login._jspService(__login.java:227)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:265)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubIm
pl.java:200)
at
weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppSe
rvletContext.java:2459)
at
weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestIm
pl.java:2039)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:139)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)--
Ture Hoefner
BEA Systems, Inc.
2590 Pearl St.
Suite 110
Boulder, CO 80302
www.bea.com

DATABASE SHUTDOWN 후 STARTUP시 ORA-7445가 발생하면서 OS PROMPT로 빠짐

제품 : ORACLE SERVER
작성날짜 : 2002-04-10
DATABASE SHUTDOWN 후 STARTUP시 ORA-7445가 발생하면서 OS PROMPT로 빠짐
=====================================================================
Server Platform : Sparc Sunw Ultra Enterprise 10000
(CPU 44개, Main Memory 9 Megabytes)
ORACLE RDBMS 7.3.3.4
ORACLE PARALLEL SERVER 환경
Sun Cluster Volume Manager 2.0
DB size : 약 1.2 Terabytes (=1200 Gigabytes)
관련 TAR : 9168089.7
조치 내역 :
1. init.ora에 다음의 event와 hidden parameter를 설정
1) event="10210 trace name context forever, level 10"
(Data Block Integrity를 check, Refer to Note 21184.1)
2) event="10211 trace name context forever, level 10"
(Data Block Integrity를 check, Refer to Note 21185.1
3) dbblock_cache_protect = true (Refer to note18144.1)
corrupt된 block이 memory에 load되는 것을 방지
2. Startup mount 후 recover database를 실행
ORA-600[6593] [26]발생 - Detection of block corruption
3. alter database open 실행
ORA-1578 file#88 block #172628 발생 하였으나
ORA-7445 없이 open됨
(문제가 되는 특정 테이블과 데이타 블럭 발견)
4. 위의 file block에 data를 확인하기 위하여 block dump를 함
"Alter session set events 'immediate trace name
blockdump level xxxxx"
( xxxxx는 Data Block Address 값의 decimal 값)
- Block dump에 있는 hexa값을 ascii로 변환
5. corrupt된 block을 제외한 data를 recover하기 위하여 새로운
tablespace를 생성 후 다음 실행.
"Insert into (TABLEA) select * from (TABLEB)
where index_column > ' '
and rowid not like 'corrupt된 row의 rowid들';"
(TABLEA는 corrupt되지 않은 row들을 제외한 나머지 row들을
포함할 새로운 테이블이고, TABLEB는 corrupt된 row들을 갖고
있었던 기존의 테이블)
6. 새로 생성한 테이블의 데이타 확인한 후 기존의 corrupt된
테이블 drop시킴
7. 새 테이블을 rename하여 기존의 테이블 이름과 같게 함.

OBIEE 11.1.1.6.2 BP1 - Excel Export every 300 row corrupt

Hi
I'm facing an issue which looks like a bug. Some reports are corrupt after export to excel.
When I remove all Header-Rows from this reports in excel then always row number 301, 601, 901... are concerned. In these rows the content of the first cells is deleted.
Has anyone discovered the same error? Otherwise I have to open a call by oracle.
Regards

Hi folks,
I've been the same problem to export table on excel.
It's interesting, because it is a pattern of blank rows. For sure, it is a bug
I looked for on supporte.oracle and found a patch "Download and apply patch 14013626 for OBIEE 11.1.1.6.0 or OBIEE 11.1.1.6.2. The only operating system currently for this patch is 64-bit Windows.".
However, this patch only works on 64-bit windows.
Does someone know another way to fix this bug for linux?

Jtable on JScrollPane get corrupted for large number of rows

Hi I have problem with vertical scroll bars of JScrollPane.
When move the scroll bar(faster) for a Jtable (with 2000 rows) the rows get
corupted.
Please let me know how can I fix this problem?

Hi,
I have just recompiled my (previously 1.3.1) application with 1.4.2 and notice the same problem. The problem starts somewhere between 1700 and 2500 rows.
Its not just the scroll bar for me - the display corrupts whereever i click the mouse on the table area.
Did you manage to diagnose??
Thanks, Dave

Using firefox 4.0.1, IBM SVC console internal server, rows with I think form data, the last row is corrupted and overlays a prior row.

IBM SVC console uses websphere. Screen data that is presented in table form last row appears to overwrite the prior row only. In other similar layouts, the last row is cut in half and cannot be seen. The prior version of Firefox 3 (something) did not have this problem. OS Windows XP Pro SP3
Firefox V3.6.17 worked fine

Seeing the same problem using the HMC with FF7 beta.

Corrupt row

Similar Messages

Maybe you are looking for