Retrieve data from a large table from ORACLE 10g

I am working with a Microsoft Visual Studio Project that requires to retrieve data from a large table from Oracle 10g database and export the data into the hard drive.
The problem here is that I am not able to connect to the database directly because of license issue but I can use a third party API to retrieve data from the database. This API has sufficient previllege/license permission on to the database to perform retrieval of data. So, I am not able to use DTS/SSIS or other tool to import data from the database directly connecting to it.
Here my approach is...first retrieve the data using the API into a .net DataTable and then dump the records from it into the hard drive in a specific format (might be in Excel file/ another SQL server database).
When I try to retrieve the data from a large table having over 13 lacs records (3-4 GB) in a data table using the visual studio project, I get an Out of memory exception.
But is there any better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?
Any help on this problem will be highly appriciated.
Thanks in advance...
-Jahedur Rahman
Edited by: Jahedur on May 16, 2010 11:42 PM

Girish...Thanks for your reply...But I am sorry for the confusions. Let me explain that...
1."export the data into another media into the hard drive."
What does it mean by this line i.e. another media into hard drive???
ANS: Sorry...I just want to write the data in a file or in a table in SQL server database.
2."I am not able to connect to the database directly because of license issue"
huh?? I never heard this question that a user is not able to connect the db because of license. What error / message you are getting?
ANS: My company uses a 3rd party application that uses ORACLE 10g. And my compnay is licensed to use the 3rd party application (APP+Database is a package) and did not purchased ORACLE license to use directly. So I will not connect to the database directly.
3.I am not sure which API is you are talking about, but i am running an application of the visual studio data grid or similar kind of controls; in which i can select (select query) as many rows as i needed; no issue.
ANS: This API is provided by the 3rd party application vendor. I can pass a query to it and it returns a datatable.
4."better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?"
ANS: As I get a system error (out of memory) when I select all rows in a datatable at a time, I wanted to retrieve the data in multiple phases.
E.g: 1 to 20,000 records in 1st phase
20,001 to 40,000 records in 2nd phase
40,001 to ...... records in 3nd phase
and so on...
Please let me know if this does not clarify your confusions... :)
Thanks...
-Jahedur Rahman
Edited by: user13114507 on May 12, 2010 11:28 PM

Similar Messages

  • Field-Symbols: How to retrieve data into an internal table from FS

    Hello All,
    I am working on field symbols.I have declared the field symbols as shown.
    FIELD-SYMBOLS: <gt_pos_data>  TYPE table,
                               <wa_pos_data> like <gt_pos_data>.
    Data: Begin of itab occurs 0,
               field1(5) type c,
               field2(10)_type c,
             end of itab.
    The FS  <gt_pos_data> has 100 fields but I need to move only two fields from this FS to my internal table itab.I am doing the following.
    loop at <gt_pos_data> assigning <wa_pos_data>.
    itab-field1 = <wa_pos_data>-field1.
    itab-field2 = <wa_pos_data>-field2.
    append itab.
    clear itab.
    endloop.
    But it is giving me an error saying "<wa_pos_data> is a table without header line and therefore has no componet FIELD1".How to achieve this requirement?
    Thanks in advance
    Sandeep

    <wa_pos_data> should be defined LIKE LINE OF <gt_pos_data>, but it will still have no structure, so you need to assign a field symbol to the component as well.
    FIELD-SYMBOLS: <gt_pos_data> TYPE table,
                               <wa_pos_data> like LINE OF <gt_pos_data>,
                               <field> type any.
    Data: Begin of itab occurs 0,
    field1(5) type c,
    field2(10)_type c,
    end of itab.
    loop at <gt_pos_data> assigning <wa_pos_data>.
    assign componet 'FIELD1' of structure <wa_pos_data> to <field>.
    if sy-subrc  = 0.
    itab-field1 = <field>.
    endif.
    assign componet 'FIELD2' of structure <wa_pos_data> to <field>.
    if sy-subrc  = 0.
    itab-field2 = <field>.
    endif.
    append itab.
    clear itab.
    endloop.
    Regards,
    Rich Heilman

  • Unable to retrieve data from a nested table in Oracle 8i from JSP

    How do i retrieve data from a nested table in Oracle 8i from my JSP code?
    When i try to execute the query , a general error is thrown.
    Kindly advice as soon as possible.

    How do i retrieve data from a nested table in Oracle 8i from my JSP code?
    When i try to execute the query , a general error is thrown.
    Kindly advice as soon as possible.

  • Insert /delete data from SAP Z table to Oracle table and opposite

    Hi,
    Can u help me write this FM from the SAP side?
    So, I have two tables ZTABLE in SAP and Oracle table ORAC.
    Let's put three columns in each of them, for example
    TEL1
    TEL2
    ADRESS
    NAME
    where TEL field is primary from ZTABLE to ORAC...
    (in FM there shoud be abap code for writing data in ZTABLE after we press some pushbutton made in sap screen painter..)
    for example, when we write new record in ZTABLE
    00
    112233
    Street 4
    Name1
    this data shoud be inserted in Oracle table ORAC.
    when we write new record in Oracle table for example
    01
    445566
    New Street
    Name2
    this data shoud be inserted in ZTABLE.
    Field TEL1 can be only of two values 01 or 02, other combination is not valid...
    I must have all data from Oracle table ORAC in ZTABLE and opposite.
    It should be the same scenario for DELETE...
    And this communication should be online between sap and table in oracle database...
    Can u help me from sap side? and give idea how to configure on oracle side??
    Thanks a lot,
    Nihad

    I dont know if we can directly connect to a oracle database ( wait for the answers from others on this )
    but in XI we have the JDBC adaptor to insert and retrieve data.
    so for the outbound from SAP the flow can be something like this (with XI in landscape):
    1) You have a screen to maintain a new entry / delete an entry
    2) On save , this record gets saved or deleted from the Ztable in SAP
    3)) In the same screen you can call a proxy class-method (generated using SPROXY transaction ) to send the record to XI.
    4) XI to format it and insert into the oracle table
    Mathews

  • SELECTing from a large table vs small table

    I posted a question few months back about teh comparison between INSERTing to a large table vs small table ( fewer number of rows ), in terms of time taken.
    The general consensus seemed to be that it would be teh same, except for teh time taken to update the index ( which will be negligible ).
    1. But now, following teh same logic, I m confused why SELECTINg from a large table should be more time taking ("expensive" ) than SELECTing from a small table.
    ( SELECTing using an index )
    My understanding of how Oracle works internally is this :
    It will first locate the ROWID from teh B-Tree that stores the index.
    ( This operation is O(log N ) based on B-Tree )
    ROWID essentially contains teh file pointer offset of teh location of the data in teh disk.
    And Oracle simply reads teh data from teh location it deduced from ROWID.
    But then the only variable I see is searching teh B-Tree, which should take O(log N ) time for comparison ( N - number of rows )
    Am I correct above.
    2. Also I read that tables are partitioned for performance reasons. I read about various partiotion mechanisms. But cannot figure out how it can result in performance improvement.
    Can somebody please help

    user597961 wrote:
    I posted a question few months back about teh comparison between INSERTing to a large table vs small table ( fewer number of rows ), in terms of time taken.
    The general consensus seemed to be that it would be teh same, except for teh time taken to update the index ( which will be negligible ).
    1. But now, following teh same logic, I m confused why SELECTINg from a large table should be more time taking ("expensive" ) than SELECTing from a small table.
    ( SELECTing using an index )
    My understanding of how Oracle works internally is this :
    It will first locate the ROWID from teh B-Tree that stores the index.
    ( This operation is O(log N ) based on B-Tree )
    ROWID essentially contains teh file pointer offset of teh location of the data in teh disk.
    And Oracle simply reads teh data from teh location it deduced from ROWID.
    But then the only variable I see is searching teh B-Tree, which should take O(log N ) time for comparison ( N - number of rows )
    Am I correct above.
    2. Also I read that tables are partitioned for performance reasons. I read about various partiotion mechanisms. But cannot figure out how it can result in performance improvement.
    Can somebody please helpIt's not going to be that simple. Before your first step (locate ROWID from index), it will first evaluate various access plans - potentially thousands of them - and choose the one that it thinks will be best. This evaluation will be based on the number of rows it anticipates having to retrieve, whether or not all of the requested data can be retrived from the index alone (without even going to the data segment), etc. etc etc. For each consideration it makes, you start with "all else being equal". Then figure there will be dozens, if not hundreds or thousands of these "all else being equal". Then once the plan is selected and the rubber meets the road, we have to contend with the fact "all else is hardly ever equal".

  • Deleting rows from very large table

    Hello,
    I need to delete rows from a large table, but not all of them, so I can't use truncate. The delete condition is based on one column, something like this:
    delete from very_large_table where col1=100;
    There's an index (valid, B-tree) on col1, but it still goes very slow. Is there any instruction which can help delete rows faster?
    Txh in adv.
    A.

    Your manager doesn't agree to your running an EXPLAIN PLAN? What is his objection? Sounds like the prototypical 'pointy-hair boss'.
    Take a look at these:
    -- do_explain.sql
    spool explain.txt
    -- do EXPLAIN PLAN on target queries with current index definitions
    truncate table plan_table
    set echo on
    explain plan for
    <insert query here>
    set echo off
    @get_explain.sql
    -- get_explain.sql
    set linesize 120
    set pagesize 70
    column operation     format a25
    column query_plan     format a35
    column options          format a15
    column object_name     format a20
    column order           format a12
    column opt           format a6
    select     lpad(' ',level) || operation "OPERATION",
         options "OPTIONS",
         decode(to_char(id),'0','COST = ' || NVL(to_char(position),'n/a'),object_name) "OBJECT NAME",
         cardinality "rows",     
         substr(optimizer,1,6) "OPT"
    from     plan_table
    start     with id = 0
    connect by prior id = parent_id
    There are probably newer, better ways, but this should work with all living versions of Oracle and is something I've had in my back pocket for several years now. It's not actually executing the query or dml in question, just running an explain plan on it.

  • How to transport the data contains in the table from development to product

    How to transport the data contains in the table from development to production.
    Please let me know ASAP.

    Hello Dilip
    Create a workbench request and add the following entries to the request:
    Object key: R3TR TABU <name of z-table>
    For this object add the following value key:
    - client-independent table: '*'
    - client-dependent table (e.g. client 100): '100*'
    See also: [SAP Network Blog: Transport Table Entries|/people/community.user/blog/2007/01/07/transport-table-entries]
    Regards
      Uwe

  • Migration from Informix 9.4 to Oracle 10g

    We have to perform a migration from Informix 9.4 to Oracle 10g. Many tables, many data and many stored procs.
    We use the Online mode and the imports finish with any errors, but it has not recovered any stored procedures or we can't view the Informix Stored Procedures in the Source Model in the workbench.
    In the importation log appears as no found procedures, but under owner informix are all procedures.
    What we can do?

    In the 10.1.0.4 version of the workbench we create an informix user who will have all the stored procedures.
    If you want these migrated as another user you can right click on the user in the oracle model and rename it for a start.
    If this is a large migration, I would recommend the following.
    1. generate the ddl for the oracle database.
    2. for the users, that will be created, move the objects to the users that you want these in.
    3. Make sure that the privileges that a particular user has is appropriate to use the data and execute the stored procedures if necessary.
    This tool is not just a one shot deal.. It will bring the informix db over to Oracle, but you need to verify the structure and the logic of the stored procedures as being correct.
    Barry

  • Migration from SQL Server 2005 to Oracle 10g

    Dear buddies,
    I need to migrate my tables with data from SQL Server 2005 to Oracle 10g. How can I do it in the best possible way?
    Please guide me.
    Nith

    Nith,
    I doubt if there is one best practice that can cover all cases of migrating data from sql server to Oracle. l generally find various factors case-by-case that make me chose one approach over the other. Without considering the business case and environment specific factors, some of the things I would like to know would be: The data types of the tables columns, table sizes, number of tables and if this is a one time process or a recurring process.
    One of the easiest (some times rather pesky) process would be:
    -Dump source data (Sql Server data) into flat files, csv or any preferred format
    -Setup a set of staging set of tables in the destination DB (Oracle in your case). You can setup the stage tables to enforce data-types, value sets and even some level of integrity constraints or any data logic or what ever you would like to filter out. Load the flat file data into a set of Oracle stage tables (Sql*Loader would be a good candidate)
    -You can always move data from these stage tables to you application tables
    You can script this whole process to run as batch or on demand or how ever you would like.
    I am sure you can google and find a bunch of proprietary as well as open source tools to do the same with little to a lot of work.
    good luck!

  • Migrating from SQL Server 2000 to Oracle 10g

    Hello,
    I am a newbie to Oracle 10g and haven't really worked with Oracle since 8i. I have been tasked to migrate all of our databases and tables to Oracle 10g. I need a little help getting started or pointed in the right direction on how to copy all that data from SQL Server to Oracle 10g. I know that Oracle has an Import/Export Utility, but don't know if it can do what I need. Can some one help point me in the direction I need?
    Thanks, Marion

    Ayub
    You need to download both the Oracle Migration Workbench core tool and the SQL Server 2000 plugin (SQLserver2k.jar). You copy the plugin into the plugin directory, after you unzip the Oracle Migration workbench main.
    The tool comes with release notes and a user guide, that explains the necessary setup and migration process.
    Donal

  • Migration from of database and app from SQL server 2005 to Oracle 10g

    Hello Every body,
    Lately, I have been requested to migrate one inhouse developed hub application from SQL server 2005 to Oracle 10g. The objective is to move application and also the database to Oracel 10g.
    Here is the current platform:
    OS Win 2008 64 bit
    JDK 1.4.2
    JBoss 3.2.5
    EJB 2.1
    My question is what kind of issues you guys see in above upgrade. As far I know, Schema and Database migration can be done by Oracle workbench. Oracle 10g supports JDK 1.4. Oracle 10g also supports EJB 2.1.
    Anything, I need to take into consideration or any risks or problems. I would like to list down all the risks, and accordingly, I am thinking of start the upgrade.
    Thanks in advance.
    Regards,
    Zeeshan Qureshi

    In general the Java/J2EE application needs work in the following categories:
    1. Connection Settings: Use Oracle JDBC drivers, create new data sources, connection pools and what not. Disable AUTOCOMMIT for Oracle JDBC Connections.
    2. For EJBs, you should regenerate the entity beans because some object names/column names might have changed in oracle.
    3. Any custom SQL that is in use in the EJBs will have to be ported to Oracle just as you would do in stored procedures and other applications.
    4. For Java front-ends, if you are using callouts to stored procedures and expecting result sets then they will require some changes. You need to modify the stored procedure call signature to include the REF CURSOR variables and process them.
    5. Changes to any SQL Server specific database functions manipulating character/date data will be required.
    6. CLOB/BLOB/XML apis are different across databases so if you are using those then focus on them as well.
    7. Retreiving Auto/generated keys from database also may need changes from what I have seen.
    Hope this helps..
    Regards
    Prakash
    NOTE: Not sure why but my posts from yesterday are not visible today. Even worse is that I can see them in one browser (FireFox) but not in Internet explorer. Crazy browser day I am having.

  • Migration from SQL SERVER 2008 to Oracle 10g issues.

    Hi ,
    I'm trying to migrate from SQL Server 2008 to Oracle 10g and I end up with some issues that I wanted to ask some info about it.
    First, I was following a tutorial
    http://st-curriculum.oracle.com/obe/db/hol08/sqldev_migration/mssqlserver/migrate_microsoft_sqlserver_otn.htm
    and and in the last version of Oracle SQL developer I downloaded, I didn't find the Load Database Capture Script Output option.
    My first question is how can I either add this option or is there a new way to load the capture script ?
    Then I use the migration wizard and after following everything, I had the migration complete message from the application.
    When I tried to open my connectionss, it tells me version 11.2 is required. Does that mean that Oracle SQL developper 3 only
    do the migration for oracle 11g? And if I tried the previous versions, it doesn't support SQL server 2008.
    What do I do in this case ?
    Thanks !

    Thank you for your reply.
    After using the wizard and made the offline migration script, in the Migration projects window I made a move data to the oracle connection I previously made and after when I tried to reopen the connection, I couldn't.
    Well I didn't retry it yet to see if I'll have the same behavior but basically this is what I think I did. When I tried to open the connection I made, it says, Oracle 11.2 is required.
    Now, I just generate target from the convert model and it says migration complete but how do I test it and I do I go to oracle and see change ?
    Do I had to create a special user in SQL Server to log to it first ? How do I logon to it ?
    Edited by: 873671 on Jul 20, 2011 7:03 AM
    Edited by: 873671 on Jul 20, 2011 7:19 AM

  • Converting SQL statements from MS server 2000 to Oracle 10g

    We are moving over from MS server 2000 to Oracle 10g as our database for Peoplesoft system.
    There are several embedded SQL statements that I need to investigate to see what needs converting.
    So far I can see a need to convert the following:
    Dates.     GetDate() to ?
    Outer joins. *= to LEFT OUTER JOIN
    Has anyone else done a similar exercise and what other functions do I need to convert?
    Thanks.

    Hello
    A quick google search (http://www.google.co.uk/search?hl=en&q=ms+sql+server+oracle+differences&spell=1)
    came up with this:
    http://dba-oracle.com/oracle_news/2005_12_16_sql_syntax_differences.htm
    There's a quite a few more sites listed.
    HTH
    David

  • Migration from MS SQL Server to Oracle 10g

    Hi,
    In our application, we are planning to refresh data every one hour from MS SQL Server to Oracle 10g. Can anyone tell me what approach can be followed?
    Thanks & Regards,
    Faizal MK

    Hello,
    migrations can be done with the Migration Workbench that is included in the SQL Developer: http://www.oracle.com/technology/tech/migration//workbench/index_sqldev_omwb.html
    But your question sounds more like a replication of data than a migration. Please read as a starter the following note in My Oracle Support (former Metalink):
    Note 283700.1: How to replicate Data between Oracle and a Foreign Datasource
    That note describes ways for the replication in both directions. Please let me know whether this is helpful for you.
    Best regards
    Wolfgang Kobarg-Sachsse

  • How to hide the data in particular table in oracle 10g

    How to hide the data in particular table in oracle 10g
    i want steps

    If its on Report u can  always hide the column - Keyfigure or Selection - Display - Hide......y do u want to have it on the report if it is to be hided in the first place?

Maybe you are looking for

  • Adding a custom button in agile web client GUI.

    Hi, Is it possible to add a custom button in agile web client GUI?. Since i am totally new in the agile world, please provide some guidelines. thanks, Uday

  • What is the command for Defragging?

    I was told that my system probably needs to be defragged. I heard that there is a Key Command for that action. Something like Com>Control>D>F. Not sure though, so does anyone know what it is? eMac   Mac OS X (10.3.9)  

  • Use of SQ01, SQ02, SQ03

    Hai All,          May i know that what is the use of SQ01 , SQ02, SQ03.How to use this ? How to create it ? Can we write our requirement in this ( addition to generated one )? Plz rely . Thanks & Regards Prashanth Message was edited by: Prashanth Kon

  • Changing PC's and syncing my Ipod.

    Recently I purchased a new PC and download Itunes to it. Now can I sync my Ipod to the new PC and get my songs on my new PC?

  • Does reader for iOS support text reflow?

    I have a number of tagged PDFs that I created using Acrobat on a Windows machine, and transferred them to my iPhone. Does Reader on iOS support text reflow so that I can increase the font and read my document? The only option I see is to zoom in, but