ETL Using Pro*C

Hi,
I'm working on a Data Warehousing environment, till now we were using Informix XPS 8.32, with application development done on ESQL/C. But now the client wants to do start with a project on Oracle (10g Release2), with Pro*C.
As I don't have any experience working on ETL application on Pro*C. The main challenges that I can foresee now are:
1. How do I extract the data from Informix databases using Pro*C(Since all the source databases are still on Informix)?
2. Does Oracle provide some Temporary / Scratch tables, that are fast enough to cater to huge no. of records (approx 20 mn rows per month) for transformation?
3. Does Oracle has some External tables by which Flat files can directly be loaded into Oracle and then transformed into Permanent tables?
4. Is there some documentation available for ETL Best Practices while developing the datamarts using Pro*C?
Appreciate if you could provide me with your valuable inputs and best practices on how to do it. A sample code would be highly appreciated.
Many many thanks in advance to all.
P.S. : The client requirement is very specific, to use Pro*C for development(no ETL tools & not much PL/SQL).
Thanks
Manish

1.
<quote>I recomend you to use ETL from Informatica, which knows to deal with Informix and also with Oracle ( this is from migrating data from Informix to Oracle)
</quote>
Sure, if money and speed is not a concern. And of course, having access to Informatica expertise (in-house or external contracts) kind of matters.
And anyway, if this is about migrating data from Informix to Oracle and not integrating two systems then purchasing any ETL tool, if not already readily available in-house, is likely overkill.
Manish: Pro*C is a precompiler ... at the end of the day one still has to compile and link a C program. So, if you know how to code C to extract data from Informix ... then write that part and then the PRO*C part will cover the Oracle side (just mix C and Pro*C code). The easiest [in this case] would be to separate the Extract (gets data from Informix into flat files) and the Loader (the PRO*C program reading flat files and writing to Oracle tables). In Unix you could even pipe the output of one executable to the input of the other. But in this day and age, unless one is a commercial ETL vendor, all this is overly complicated.
2.
<quote>The problem with Pro*C is that is too slow, I have made a test on a 2 milion record table and the result was that OCI was 20 more times faster.</quote>
I doubt that very much.
I doubt not what you have noticed ... only the conclusion you've drawn out of that observation. I can write a PL/SQL procedure inserting rows one-by-one and compare that with a PRO*C program doing array processing ... it will be slower, but that proves what? The fastest [and easiest] way to get data loaded into Oracle (when applicable) is sql*loader in direct path.
<quote>A better Idea is to tell your client that is wrong chosing Pro*C</quote>
What is wrong is the customer rejecting the most useful mix of Oracle technologies for this custom (since they rejected the [expensive] commercial tools) ETL job: sql*loader, sql, pl/sql ... what they've chosen instead is almost irrelevant. The only justification would be if they are indeed a shop specializing in Pro*C (in which case they asked the wrong person to code this) ... though Pro*C (hence Oracle) experts rejecting PL/SQL is kind of a contradiction.

Similar Messages

  • ORA-01458 error while using Pro*C to invoke PL/SQL procedure, pls help

    I am using Pro*C (Oracle 10g on Itanium platform) to invoke PL/SQL procedure to read RAW data from database, but always encoutered ORA-01458 error message.
    Here is the snippet of Pro*C code:
    typedef struct dataSegment
         unsigned short     len;
         unsigned char     data[SIZE_DATA_SEG];
    } msg_data_seg;
    EXEC SQL TYPE msg_data_seg IS VARRAW(SIZE_DATA_SEG);
         EXEC SQL BEGIN DECLARE SECTION;
              unsigned short qID;
              int rMode;
              unsigned long rawMsgID;
              unsigned long msgID;
              unsigned short msgType;
              unsigned short msgPriority;
              char recvTime[SIZE_TIME_STRING];
              char schedTime[SIZE_TIME_STRING];
              msg_data_seg dataSeg;
              msg_data_seg dataSeg1;
              msg_data_seg dataSeg2;
              short     indSeg;
              short     indSeg1;
              short     indSeg2;
         EXEC SQL END DECLARE SECTION;
         qID = q_id;
         rMode = (int)mode;
         EXEC SQL EXECUTE
              BEGIN
                   SUMsg.read_msg (:qID, :rMode, :rawMsgID, :msgID, :msgType, :msgPriority, :recvTime,
                        :schedTime, :dataSeg:indSeg, :dataSeg1:indSeg1, :dataSeg2:indSeg2);
              END;
         END-EXEC;
         // Check PL/SQL execute result, different from SQL
         // Only 'sqlcode' and 'sqlerrm' are always set
         if (sqlca.sqlcode != 0)
              if (sqlca.sqlcode == ERR_QUEUE_EMPTY)          // Queue empty
                   throw q_eoq ();
              char msg[513];                                        // Other errors
              size_t msg_len;
              msg_len = sqlca.sqlerrm.sqlerrml;
              strncpy (msg, sqlca.sqlerrm.sqlerrmc, msg_len);
              msg[msg_len] = '\0';
              throw db_error (string(msg), sqlca.sqlcode);
    and here is the PL/SQL which is invoked:
    SUBTYPE VarChar14 IS VARCHAR2(14);
    PROCEDURE read_msg (
         qID          IN     sumsg_queue_def.q_id%TYPE,
         rMode          IN     INTEGER,
         raw_msgID     OUT     sumsg_msg_data.raw_msg_id%TYPE,
         msgID          OUT sumsg_msg_data.msg_id%TYPE,
         msgType          OUT sumsg_msg_data.type%TYPE,
         msgPrior     OUT sumsg_msg_data.priority%TYPE,
         msgRecv          OUT VarChar14,
         msgSched     OUT VarChar14,
         msgData          OUT sumsg_msg_data.msg_data%TYPE,
         msgData1     OUT sumsg_msg_data.msg_data1%TYPE,
         msgData2     OUT sumsg_msg_data.msg_data2%TYPE
    ) IS
    BEGIN
         IF rMode = 0 THEN
              SELECT raw_msg_id, msg_id, type, priority, TO_CHAR(recv_time, 'YYYYMMDDHH24MISS'),
                   TO_CHAR(sched_time, 'YYYYMMDDHH24MISS'), msg_data, msg_data1, msg_data2
                   INTO raw_msgID, msgID, msgType, msgPrior, msgRecv, msgSched, msgData, msgData1, msgData2
                   FROM (SELECT * FROM sumsg_msg_data WHERE q_id = qID AND status = 0 ORDER BY sched_time, raw_msg_id)
                   WHERE ROWNUM = 1;
         ELSIF rMode = 1 THEN
              SELECT raw_msg_id, msg_id, type, priority, TO_CHAR(recv_time, 'YYYYMMDDHH24MISS'),
                   TO_CHAR(sched_time, 'YYYYMMDDHH24MISS'), msg_data, msg_data1, msg_data2
                   INTO raw_msgID, msgID, msgType, msgPrior, msgRecv, msgSched, msgData, msgData1, msgData2
                   FROM (SELECT * FROM sumsg_msg_data WHERE q_id = qID AND status = 0 ORDER BY recv_time, raw_msg_id)
                   WHERE ROWNUM = 1;
         ELSE
              SELECT raw_msg_id, msg_id, type, priority, TO_CHAR(recv_time, 'YYYYMMDDHH24MISS'),
                   TO_CHAR(sched_time, 'YYYYMMDDHH24MISS'), msg_data, msg_data1, msg_data2
                   INTO raw_msgID, msgID, msgType, msgPrior, msgRecv, msgSched, msgData, msgData1, msgData2
                   FROM (SELECT * FROM sumsg_msg_data WHERE q_id = qID AND status = 0 ORDER BY priority, raw_msg_id)
                   WHERE ROWNUM = 1;
         END IF;
         UPDATE sumsg_msg_data SET status = 1 WHERE raw_msg_id = raw_msgID;
    EXCEPTION
         WHEN NO_DATA_FOUND THEN
              raise_application_error (-20102, 'Queue empty');
    END read_msg;
    where sumsg_msg_data.msg_data%TYPE, sumsg_msg_data.msg_data1%TYPE and sumsg_msg_data.msg_data2%TYPE are all defined as RAW(2000). When I test the PL/SQL code seperately, everything is ok, but if I use the Pro*C code to read, the ORA-01458 always happen, unless I change the SIZE_DATA_SEG value to 4000, then it passes, and the result read out also seems ok, either the bigger or smaller value will encounter ORA-01458.
    I think the problem should happen between the mapping from internal datatype to external VARRAW type, but cannot understand why 4000 bytes buffer will be ok, is it related to some NLS_LANG settings, anyone can help me to resolve this problme, thanks a lot!

    It seems that I found the way to avoid this error. Now each time before I read RAW(2000) data from database, i initialize the VARRAW.len first, set its value to SIZE_DATA_SEG, i.e., the outside buffer size, then the error disappear.
    Oracle seems to need this information to handle its data mapping, but why output variable also needs this initialization, cannot precompiler get this from the definition of VARRAW structure?
    Anyone have some suggestion?

  • Using Pro*C on Linux Oracle 8.1.6.0 on RH 6.2

    Whe have some problems using Pro*C:
    the program runs ok on all Oracle versions, including Oracle 8.1.6.0 on DEC OSF, but various sql errors are encountered on Linux Red Hat 6.2:
    1) "ORA-01458: invalid length inside variable character string" for this code:
    EXEC SQL BEGIN DECLARE SECTION;
    varchar I_LANGLB [4];
    short O_NOLBLB ;
    varchar O_LIBELB [26];
    EXEC SQL END DECLARE SECTION;
    EXEC SQL INCLUDE SQLCA.H;
    EXEC SQL WHENEVER SQLERROR GO TO MAJ_RESULT;
    EXEC SQL WHENEVER NOT FOUND CONTINUE;
    EXEC SQL DECLARE C0 CURSOR FOR
    SELECT LBL.NOLBLB, LBL.LIBELB
    FROM LBL
    WHERE LBL.EDITLB = 'MON' AND LBL.LANGLB = :I_LANGLB;
    strcpy (I_LANGLB.arr, "fra");
    I_LANGLB.len = 3;
    EXEC SQL OPEN C0;
    for ( ; ; )
    EXEC SQL WHENEVER NOT FOUND DO break;
    EXEC SQL FETCH C0 INTO :O_NOLBLB, :O_LIBELB;
    EXEC SQL CLOSE C0;
    2) with Dynamic Sql: "ORA-01007: variable not in select list"
    SELECT
    nvl(MODEME, ''),
    nvl(NBANME, 0),
    nvl(BASEME, '0'),
    nvl(PRORME,'0'),
    nvl(EVALME, '0'),
    nvl(DECOME, '0') ,
    nvl(CDDAME, '0'),
    nvl(CDMIME, '0'),
    nvl(TXMIME, 0),
    nvl(CDMAME, '0'),
    nvl(TXMAME, 0),
    nvl(CDTXME, '0'),
    nvl(CDSUME, '0'),
    nvl(TXSUME, 0),
    nvl(DUMIME, 0),
    nvl(MTVNME, 0),
    nvl(NOANML, 0),
    nvl(TAUXML, 0),
    DESIME
    FROM MET, LME
    WHERE MODEME = MODEML
    AND nvl(INLBME,'1')='1'
    ORDER BY MODEME,NOANML
    [ or
    ORDER BY 1,17 ]
    In both cases,
    We use the following precompiling options:
    include=/oracle/OraHome1/precomp/lib ireclen=132 oreclen=132 sqlcheck=syntax parse=partial select_error=no char_map=VARCHAR2 mode=ORACLE unsafe_null=yes
    dbms=V8
    Could someone help ?
    Thanks ...

    My answer is only for 1. problem (about ORA-01458)
    I think that you use declaration for cursor C0 with a varchar
    variable before you ininitialize length (member .len of varchar
    structure) of this variable.
    It's famous that many errors come from uninitialized varchar
    variables in Pro*C.

  • How can I use Pro Tools with Fire Wire interface and Fire Wire external HD

    If Mac Book Pro has only one Fire Wire jack and I want to use Pro Tools with an M Audio 1814 Firewire interface and an external Fire Wire HD how do I get around the fact that the Mac Book Pro has only one fire wire jack?
    Pro Tools says that they DO NOT recommend using USB 2 Hard Drives as recording drives.
    If I plug my External HD into the back of the 1814 it is way too slow to use, plus, I don't know why, Pro Tools is telling me that my External HD is "not a valid audio volume". It is a MAC OS Extended (Journaled) external Firewire HD (G-TECH G-DRIVE)
    Stuck...
    Message was edited by: Mezcla

    You could consider getting a firewire hub or get an expresscard 34 adapter with a firewire port.
    I personally use a Belkin Expresscard adapter that gives me an extra firewire port as well as two extra USB 2.0 ports.
    Good luck.

  • I use Pro Logic 9 and see that to upgrade to Pro Logic X I need to buy the full package again. Does apple offer special conditions for their Apple clients ? Are users of apple care protection plan better served ?

    I use Pro Logic 9.1.8.  and see that to upgrade to Pro Logic X I need to buy the full package again. Does apple offer special conditions for their Apple clients ? Are users of apple care protection plan better served ?

    Old saying applies here most of the time......
    "If it ain't broken, don't fix it"
    I choose to use LP9 with Snow Leopard myself (10.6.8 and 9.1.6) for most of my commercial work simply because I find that combo very stable and provides the best overall performance for my setups here... but I am also using LPX with Mavericks as part of my client support and that works fine too although with a drop in performance due to the heavier demands made on the hardware by the newer OS X and LPX graphic routines for example..

  • Using pro* c in forms 6i

    Hi,
    Can i use pro* c program in forms 6i ?
    if yes , then how ?
    thank you..

    Hello,
    The answer is : NO.
    Pro*C is linked to a C application, so is part of the .exe programme generated by your favourite C compiler.
    Francois

  • Write to file using pro*C

    I have a procedure that would retrieve some table info based on which computes the sum of transactions for different types of transactions. I want to prepare a summary transaction report that would give the total transactions for each transaction type. Can someone sugest a pro*C proram to perform this task. I'd like to format the report using pro*C. I've never done a pro*C before and would like to take your help in this regards.
    Thanks in advance

    http://asktom.oracle.com/pls/ask/f?p=4950:8:::::F4950_P8_DISPLAYID:459020243348

  • How to use Pro*C

    HI!
    I have a problem with using Pro*c.
    I have to store values in a database.
    I am able to do that with fixes numbers, but not with values.
    My code is:
    EXEC SQL EXECUTE
    BEGIN
    INSERT INTO XML (art_id,titel,fachgebiet,arttext) values (1,'der Titel',' Imunologie ','Das ist der Text2');
    END;
    END-EXEC;
    How can I store varying values in a database?
    Please help me.
    Thank you
    null

    In the basic C code you to declare variable that needs to be passed to oracle with in
    EXEC SQL BEGIN DECLARE SECTION
    VARCHAR empname[11];
    VARCHAR jobtype[9];
    VARCHAR hired[9];
    int salary;
    int dept;
    int worked_longer;
    int higher_sal;
    int total_in_dept;
    VARCHAR uid[20];
    VARCHAR pwd[20];
    EXEC SQL END DECLARE SECTION;
    The datatype supported are basic C datatypes and varchar etc.
    Then you can assign values to these variables in your C code and can reference these variable in a PL/SQL block prefixed by : , this are called host variables .
    Then your code will look like
    EXEC SQL EXECUTE
    BEGIN
    INSERT INTO XML (art_id,titel,fachgebiet,arttext) values (:art_id,:title,:fachgebiet ,:arttext );
    END;
    END-EXEC;
    Hope i have answered your question .
    - Bala
    null

  • App using Pro*C 8.1.7 has problem with remote 9i database

    Someone must have encountered and overcome this by now:
    A very simple application using Pro*C 8.1.7 connects properly to remote 8i databases, both at my development site and at my customers.
    When I attempt to connect to a 9i (9.2.0.1.0) database, everything starts out great: the database OPEN works, and I can execute an ALTER SESSION, SET AUTOCOMMIT ON, INSERT, and one SELECT. The second SELECT blows up in ORASQL8.dll, apparently branching to 0.
    Looks like some unincluded dll, or maybe a buffer not long enough. I don't have Pro*C for 9i, and while I don't mind having to get it and relink this, I can't believe that everyone's 8i applications have to be relinked to run against a higher level database.
    Any ideas or experiences will be greatly appreciated.

    we had the problem and this fixed it.
    try connecting to remote client using dedicated server.
    some firewalls block the outgoing ports while still leaving (incoming) 1521 open.
    net8 assistant>local>profile>general(dropdown)>routing(tab)
    checkbox (check)always use dedicated server.
    reply if helps
    CW

  • Oracle 10 Million records insert using Pro c

    Hi,
         As i am new to Oracle 10G Pro c and SQL Loader, i would like to know few informations regarding the same.
    My requirement is this that, i need to read a 20GB file (20Million Lines) line by line and convert the line into database records with few data manipulation and insert
    into a Oracle 10G database table.
         I read some articles and it says Pro C is faster than SQL Loader in performance (fast insertion). And also Pro C talks to
    the oracle directly and it puts the data pages directly into Oracle but not through SQL Engine or Parser.
         Even in the Pro c samples, i have seen a For loop to insert mulitple records. Will each insertion cost
         more time ?
         Is there any bulk insert program on Pro C like 10 Million rows at a shot ?
         Or Pro c can do upload of a file data into Oracle database table ?
         If any one already posted this query means please inform me the thread number or id
    Thank you,
    Ganesh
    Edited by: user12165645 on Nov 10, 2009 2:06 AM

    Alex Nuijten wrote:
    Personally I would go for either an External Table or SQL*Loader, mainly because I've never used Pro*C.. ;)
    And so far I never needed to resort to other option because of poor performance.I fully agree. Also we are talking about "only" 10 mill rows. This will take some time, but probably not more than 30 min, max. 2 hours depending on many factors including storage, network and current database activities. I've seen systems where such a load took only a few minutes.
    I guess if there is a difference between Pro*C and external tables I would still opt for external tables, because they are so much easier to use.

  • Using Pro*C with Turbo C Environment(16-bit Application)

    I have a 16-bit application developed using Turbo C Compiler. I want to provide a Oracle database using Pro*C Interface. When i installed Oracle 8i and Proc 8.1.5 and when i tried to build my project it gives the following error
    "Linking Error: _sqlcxt......." what is this???
    As a testing purpose, when i run the same program with Visual C++ environmet (cl.exe compiler), i could perfectly run the application and the data storage/retrieval successful
    Is it the problem with the Version clashes??
    Which Version of Pro*C can i use with Tuboc 3.0 version???
    whether it support for Turboc???
    Please advice!!!!
    Thanks
    Regards,
    Shibu
    Ace Technosoft
    Mumbai

    When I said I doesn't work- I meant It started and
    hanged after printing
    "Waiting for...". It doesn't promt any window or
    anything.Then the bat file is still executing. My guess is that it might be writing text to the standard output device, which nothing is consuming, so it is blocked on I/O waiting for you to consume the output by reading it.
    Hmmm. There was an article on www.javaworld.com about that which I was going to point you to, but it seems to have been archived. Drat.
    Well I found it anyway, but not by searching JavaWorld's site - but thru google instead. I guess JavaWorld has removed it from its searchable index for some reason.
    http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html

  • Error using PRO C++

    Folks, im making a program using PRO C++, but i receiving this
    error:
    Error occured..err_handler() -> ORA-01480: trailing null missing
    from STR bind value....
    I think it is caused by a OUT variable....

    No need to specify the length if the data type is number and pointing to Long.
    You can directly map the values using INTO clause.

  • Hello, is there a way to redact a word or item from a pdf using adobe pro X the same way you can if you convert a word doc using pro X on a pc?

    hello, is there a way to redact a word or item from a pdf using adobe pro X the same way you can if you convert a word doc using pro X on a pc?

    If the document is not a scanned image or protected from editing then you should be able to edit it. I would have to guess you have a scan and when you converted to Word, you ran OCR (Optical Character Recognition) on it converting the scanned image to live text.
    This is the forum for the free Adobe Reader which can not edit or redact.

  • How to use Pro*COBOL with RMCobol

    Hi,
    I try use Pro*Cobol with RMCobol compiler.
    But I have problem to use oraSQL9.LIB y with runcobol (runtime).
    Thank for any help.

    I try use Pro*Cobol with RMCobol compiler.
    But I have problem to use oraSQL9.LIB y with runcobol
    (runtime).You will be able to do it with the new version of RM.
    This is due very soon, and you will need the new compiler, the new runtime, and a small C wrapper that I supply to be able to do it.
    If you already have Version 7 of the runtime you can also use OCI call's as of now, this also with a C wrapper I supply (this for windows, UnixWare 7.xx, OpenUnix8, and Linux (redhat at the moment, should work with most of others).
    I can be contacted at [email protected]
    regards
    Frederico Fonseca

  • I would like to share midi tracks with my friend who uses Pro Tools 11 on a PC.  Can this be done?

    I would like to share midi tracks with my friend who uses Pro Tools on a PC.  Is this possible?  How do we send each other tracks?  Will we be able to make changes to the tracks and send them back?

    It looks like it depends which G5, OS, & which Avid HW you have, but according to these sites, v6.9 to v7.4 maybe...
    http://archive.digidesign.com/compato/osx/692/hd/g5.cfm
    http://archive.digidesign.com/compato/osx/hd/g5.cfm
    http://www.macworld.com/article/134068/2008/06/protools.html
    Might contact them & ask...
    http://www.avid.com/US/contact/audio

Maybe you are looking for