Problem: load SQL Server 'varbinary' data

Question

Source database: SQL Server 2000
OS: Windows 2003 (SP1)
Oracle: 10g (R2)
Datatype Mapping: SQL Server ‘varbinary(1000)’ to oracle ‘BLOB’
With the help of OMWB, I created the oracle database schema for a SQL Server DB (offline capture). I have problems when I tried to populate the (destination) database with a table with ‘varbinary(1000)’ datatype:
PlateX.expID (as on http://www.sdss.org.uk/mybestdr5/en/help/browser/description.asp?n=PlateX&t=U) and the with original data is like this: http://dsg.port.ac.uk/~hx/research/sdss/logs/SelectExpidFromPlatex
The PLATEX.ctl file (generated by OMWB) is like this (http://dsg.port.ac.uk/~hx/research/sdss/logs/PLATEX.ctl):
load data
infile 'PLATEX.dat' "str '<er>'"
into table PLATEX
fields terminated by '<ec>'
trailing nullcols
<SKIP>
EXPID)
I failed to load the data with the sql_load_script.sh script, and the log file is like this:
<SKIP>
EXPID NEXT * CHARACTER
Terminator string : '<ec>'
value used for ROWS parameter changed from 64 to 12
Record 1: Rejected - Error on table PLATEX, column EXPID.
Field in data file exceeds maximum length
<SKIP>
Record 60: Rejected - Error on table PLATEX, column EXPID.
Field in data file exceeds maximum length
MAXIMUM ERROR COUNT EXCEEDED - Above statistics reflect partial run.
Table PLATEX:
9 Rows successfully loaded.
51 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Space allocated for bind array: 238392 bytes(12 rows)
Read buffer bytes: 1048576
Total logical records skipped: 0
Total logical records read: 60
Total logical records rejected: 51
Total logical records discarded: 0
Any help would be extremely appreciated.
Helen
P.S. previously, I mapped: SQL Server ‘varbinary(1000)’ to ‘RAW(1000)’ in Oracle but failed to populate the data (http://dsg.port.ac.uk/~hx/research/sdss/logs/PLATEX.log).

Answer 1

Hi Emile,
regading extracting data from MSSQL with OWB on Unix platform (using Generic Connectivity):
Re: SQLServer access from AIX Warehouse builder
Re: OWB on Solaris Connectivity with SQL SERVER on Windows
We want to load aprox. 100 million records a day.
I've read some articles about this and the advise was to dump the data from SQL Server to files and load the files with OWB.100 million records per day is not a problem for daily extracting from MSSQL Server if you have 1-2 hour.
In my opinion dumping to text file is a bad practice and is unnecessary if customer don't have special requirements (for example for security reason).
SQL Server source table and a Oracle target table without any difficult transformationsIn my opninion the best way process data from MSSQL is to extract data to staging area (schema) on Oracle DB with mappings as simple as possible (ONLY filters, without any join), and most of data processing prefom in Staging area or during moving from staging to DWH.
Also look at OWB user guide (how to use Generic Connectivity in OWB)
http://download.oracle.com/docs/cd/E11882_01/owb.112/e10582/loading_ms_data.htm#i1064950
Regards,
Oleg
Edited by: added link to OWB doc with description of using Generic Connectivity