LARGE TABLE IN ORACLE

How could i migrate large tables multi-million row from Oracle 8i to 9i.Pls explain me the process

Vijay,
this is the wrong forum for this question. Ask yourself these questions though:
Is this an insitu upgrade, hence there will be data migration scripts as part of the upgrade, otherwise, there may be transportable tablespaces you can use.
Are the machines disparate, Is the data temporal in nature, can you subsection the datamove, holding back on the volatile data until you need to switch over?

Similar Messages

Retrieve data from a large table from ORACLE 10g

I am working with a Microsoft Visual Studio Project that requires to retrieve data from a large table from Oracle 10g database and export the data into the hard drive.
The problem here is that I am not able to connect to the database directly because of license issue but I can use a third party API to retrieve data from the database. This API has sufficient previllege/license permission on to the database to perform retrieval of data. So, I am not able to use DTS/SSIS or other tool to import data from the database directly connecting to it.
Here my approach is...first retrieve the data using the API into a .net DataTable and then dump the records from it into the hard drive in a specific format (might be in Excel file/ another SQL server database).
When I try to retrieve the data from a large table having over 13 lacs records (3-4 GB) in a data table using the visual studio project, I get an Out of memory exception.
But is there any better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?
Any help on this problem will be highly appriciated.
Thanks in advance...
-Jahedur Rahman
Edited by: Jahedur on May 16, 2010 11:42 PM

Girish...Thanks for your reply...But I am sorry for the confusions. Let me explain that...
1."export the data into another media into the hard drive."
What does it mean by this line i.e. another media into hard drive???
ANS: Sorry...I just want to write the data in a file or in a table in SQL server database.
2."I am not able to connect to the database directly because of license issue"
huh?? I never heard this question that a user is not able to connect the db because of license. What error / message you are getting?
ANS: My company uses a 3rd party application that uses ORACLE 10g. And my compnay is licensed to use the 3rd party application (APP+Database is a package) and did not purchased ORACLE license to use directly. So I will not connect to the database directly.
3.I am not sure which API is you are talking about, but i am running an application of the visual studio data grid or similar kind of controls; in which i can select (select query) as many rows as i needed; no issue.
ANS: This API is provided by the 3rd party application vendor. I can pass a query to it and it returns a datatable.
4."better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?"
ANS: As I get a system error (out of memory) when I select all rows in a datatable at a time, I wanted to retrieve the data in multiple phases.
E.g: 1 to 20,000 records in 1st phase
20,001 to 40,000 records in 2nd phase
40,001 to ...... records in 3nd phase
and so on...
Please let me know if this does not clarify your confusions... :)
Thanks...
-Jahedur Rahman
Edited by: user13114507 on May 12, 2010 11:28 PM

Partioned tables in Oracle 10

I would like to get some feedback and guidance on whether there are any advantages to the implementing partitioning on very large tables within Oracle for the purpose of webIntelligence reporting in a Business Objects XI R2 with SP5 running on AIX 5.3.7 environment. Has this been done by any clients and was reporting performance improved.

No reply was received from this post however on site we experimented with this and found it to be enormosusly beneficial fro system performance.

Pagination query help needed for large table - force a different index

I'm using a slight modification of the pagination query from over at Ask Tom's: [http://www.oracle.com/technology/oramag/oracle/07-jan/o17asktom.html]
Mine looks like this when fetching the first 100 rows of all members with last name Smith, ordered by join date:
SELECT members.*
FROM members,
    SELECT RID, rownum rnum
    FROM
        SELECT rowid as RID
        FROM members
        WHERE last_name = 'Smith'
        ORDER BY joindate
    WHERE rownum <= 100
WHERE rnum >= 1
         and RID = members.rowidThe difference between this and the one at Ask Tom's is that my innermost query just returns the ROWID. Then in the outermost query we join the ROWIDs returned to the members table, after we have pruned the ROWIDs down to only the chunk of 100 we want. This makes it MUCH faster (verifiably) on our large tables, as it is able to use the index on the innermost query (well... read on).
The problem I have is this:
SELECT rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindateThis will use the index for the predicate column (last_name) instead of the unique index I have defined for the joindate column (joindate, sequence). (Verifiable with explain plan). It is much slower this way on a large table. So I can hint it using either of the following methods:
SELECT /*+ index(members, joindate_idx) */ rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindate
SELECT /*+ first_rows(100) */ rowid as RID
FROM members
WHERE last_name = 'Smith'
ORDER BY joindateEither way, it now uses the index of the ORDER BY column (joindate_idx), so now it is much faster as it does not have to do a sort (remember, VERY large table, millions of records). So that seems good. But now, on my outermost query, I join the rowid with the meaningful columns of data from the members table, as commented below:
SELECT members.*      -- Select all data from members table
FROM members,           -- members table added to FROM clause
    SELECT RID, rownum rnum
    FROM
        SELECT /*+ index(members, joindate_idx) */ rowid as RID   -- Hint is ignored now that I am joining in the outer query
        FROM members
        WHERE last_name = 'Smith'
        ORDER BY joindate
    WHERE rownum <= 100
WHERE rnum >= 1
        and RID = members.rowid           -- Merge the members table on the rowid we pulled from the inner queriesOnce I do this join, it goes back to using the predicate index (last_name) and has to perform the sort once it finds all matching values (which can be a lot in this table, there is high cardinality on some columns).
So my question is, in the full query above, is there any way I can get it to use the ORDER BY column for indexing to prevent it from having to do a sort? The join is what causes it to revert back to using the predicate index, even with hints. Remove the join and just return the ROWIDs for those 100 records and it flies, even on 10 million records.
It'd be great if there was some generic hint that could accomplish this, such that if we change the table/columns/indexes, we don't need to change the hint (the FIRST_ROWS hint is a good example of this, while the INDEX hint is the opposite), but any help would be appreciated. I can provide explain plans for any of the above if needed.
Thanks!

Lakmal Rajapakse wrote:
OK here is an example to illustrate the advantage:
SQL> set autot traceonly
SQL> select * from (
2 select a.*, rownum x from
3 (
4 select a.* from aoswf.events a
5 order by EVENT_DATETIME
6 ) a
7 where rownum <= 1200
8 )
9 where x >= 1100
10 /
101 rows selected.
Execution Plan
Plan hash value: 3711662397
| Id | Operation                      | Name       | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT               |            | 1200 |   521K|   192   (0)| 00:00:03 |
|* 1 | VIEW                          |            | 1200 |   521K|   192   (0)| 00:00:03 |
|* 2 |   COUNT STOPKEY                |            |       |       |            |          |
|   3 |    VIEW                        |            | 1200 |   506K|   192   (0)| 00:00:03 |
|   4 |     TABLE ACCESS BY INDEX ROWID| EVENTS     |   253M|    34G|   192   (0)| 00:00:03 |
|   5 |      INDEX FULL SCAN           | EVEN_IDX02 | 1200 |       |     2   (0)| 00:00:01 |
Predicate Information (identified by operation id):
1 - filter("X">=1100)
2 - filter(ROWNUM<=1200)
Statistics
0 recursive calls
0 db block gets
443 consistent gets
0 physical reads
0 redo size
25203 bytes sent via SQL*Net to client
281 bytes received via SQL*Net from client
8 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
101 rows processed
SQL>
SQL>
SQL> select * from aoswf.events a, (
2 select rid, rownum x from
3 (
4 select rowid rid from aoswf.events a
5 order by EVENT_DATETIME
6 ) a
7 where rownum <= 1200
8 ) b
9 where x >= 1100
10 and a.rowid = rid
11 /
101 rows selected.
Execution Plan
Plan hash value: 2308864810
| Id | Operation                   | Name       | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT            |            | 1200 |   201K|   261K (1)| 00:52:21 |
|   1 | NESTED LOOPS               |            | 1200 |   201K|   261K (1)| 00:52:21 |
|* 2 |   VIEW                      |            | 1200 | 30000 |   260K (1)| 00:52:06 |
|* 3 |    COUNT STOPKEY            |            |       |       |            |          |
|   4 |     VIEW                    |            |   253M| 2895M|   260K (1)| 00:52:06 |
|   5 |      INDEX FULL SCAN        | EVEN_IDX02 |   253M| 4826M|   260K (1)| 00:52:06 |
|   6 |   TABLE ACCESS BY USER ROWID| EVENTS     |     1 |   147 |     1   (0)| 00:00:01 |
Predicate Information (identified by operation id):
2 - filter("X">=1100)
3 - filter(ROWNUM<=1200)
Statistics
8 recursive calls
0 db block gets
117 consistent gets
0 physical reads
0 redo size
27539 bytes sent via SQL*Net to client
281 bytes received via SQL*Net from client
8 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
101 rows processed
Lakmal (and OP),
Not sure what advantage you are trying to show here. But considering that we are talking about pagination query here and order of records is important, your 2 queries will not always generate output in same order. Here is the test case:
SQL> select * from v$version ;
BANNER
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE     10.2.0.1.0     Production
TNS for Linux: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production
SQL> show parameter optimizer
NAME                                 TYPE        VALUE
optimizer_dynamic_sampling           integer     2
optimizer_features_enable            string      10.2.0.1
optimizer_index_caching              integer     0
optimizer_index_cost_adj             integer     100
optimizer_mode                       string      ALL_ROWS
optimizer_secure_view_merging        boolean     TRUE
SQL> show parameter pga
NAME                                 TYPE        VALUE
pga_aggregate_target                 big integer 103M
SQL> create table t nologging as select * from all_objects where 1 = 2 ;
Table created.
SQL> create index t_idx on t(last_ddl_time) nologging ;
Index created.
SQL> insert /*+ APPEND */ into t (owner, object_name, object_id, created, last_ddl_time) select owner, object_name, object_id, created, sysdate - dbms_random.value(1, 100) from all_objects order by dbms_random.random;
40617 rows created.
SQL> commit ;
Commit complete.
SQL> exec dbms_stats.gather_table_stats(user, 'T', cascade=>true);
PL/SQL procedure successfully completed.
SQL> select object_id, object_name, created from t, (select rid, rownum rn from (select rowid rid from t order by created desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
OBJECT_ID OBJECT_NAME                    CREATED
     47686 ALL$OLAP2_JOIN_KEY_COLUMN_USES 28-JUL-2009 08:08:39
     47672 ALL$OLAP2_CUBE_DIM_USES        28-JUL-2009 08:08:39
     47681 ALL$OLAP2_CUBE_MEASURE_MAPS    28-JUL-2009 08:08:39
     47682 ALL$OLAP2_FACT_LEVEL_USES      28-JUL-2009 08:08:39
     47685 ALL$OLAP2_AGGREGATION_USES     28-JUL-2009 08:08:39
     47692 ALL$OLAP2_CATALOGS             28-JUL-2009 08:08:39
     47665 ALL$OLAPMR_FACTTBLKEYMAPS      28-JUL-2009 08:08:39
     47688 ALL$OLAP2_DIM_LEVEL_ATTR_MAPS 28-JUL-2009 08:08:39
     47689 ALL$OLAP2_DIM_LEVELS_KEYMAPS   28-JUL-2009 08:08:39
     47669 ALL$OLAP9I2_HIER_DIMENSIONS    28-JUL-2009 08:08:39
     47666 ALL$OLAP9I1_HIER_DIMENSIONS    28-JUL-2009 08:08:39
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     37534 com/sun/mail/smtp/SMTPMessage 06-FEB-2010 03:46:14
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     15752 /2f494dce_JDWPThreadReference 06-FEB-2010 03:09:31
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     37534 com/sun/mail/smtp/SMTPMessage 06-FEB-2010 03:46:14
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     15752 /2f494dce_JDWPThreadReference 06-FEB-2010 03:09:31
11 rows selected.
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid order by last_ddl_time desc ;
OBJECT_ID OBJECT_NAME                    LAST_DDL_TIME
     37534 com/sun/mail/smtp/SMTPMessage 06-FEB-2010 03:46:14
     13133 oracle/jdbc/driver/OracleLog$3 06-FEB-2010 03:45:44
     11749 /b9fe5b99_OraRTStatementComman 06-FEB-2010 03:43:49
     42266 SI_GETCLRHSTGRFTR              06-FEB-2010 03:40:20
     16695 /2940a364_RepIdDelegator_1_3   06-FEB-2010 03:38:17
     36539 sun/io/ByteToCharMacHebrew     06-FEB-2010 03:28:57
     26815 /7a628fb8_DefaultHSBChooserPan 06-FEB-2010 03:26:55
     14044 /d29b81e1_OldHeaders           06-FEB-2010 03:12:12
     36145 /4e492b6f_SerProfileToClassErr 06-FEB-2010 03:11:09
     12920 /25f8f3a5_BasicSplitPaneUI     06-FEB-2010 03:11:06
     15752 /2f494dce_JDWPThreadReference 06-FEB-2010 03:09:31
11 rows selected.
SQL> set autotrace traceonly
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid order by last_ddl_time desc
2 ;
11 rows selected.
Execution Plan
Plan hash value: 44968669
| Id | Operation                       | Name | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT                |       | 1200 | 91200 |   180   (2)| 00:00:03 |
|   1 | SORT ORDER BY                  |       | 1200 | 91200 |   180   (2)| 00:00:03 |
|* 2 |   HASH JOIN                     |       | 1200 | 91200 |   179   (2)| 00:00:03 |
|* 3 |    VIEW                         |       | 1200 | 30000 |    98   (0)| 00:00:02 |
|* 4 |     COUNT STOPKEY               |       |       |       |            |          |
|   5 |      VIEW                       |       | 40617 |   475K|    98   (0)| 00:00:02 |
|   6 |       INDEX FULL SCAN DESCENDING| T_IDX | 40617 |   793K|    98   (0)| 00:00:02 |
|   7 |    TABLE ACCESS FULL            | T     | 40617 | 2022K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   2 - access("T".ROWID="T1"."RID")
   3 - filter("RN">=1190)
   4 - filter(ROWNUM<=1200)
Statistics
          1 recursive calls
          0 db block gets
        348 consistent gets
          0 physical reads
          0 redo size
       1063 bytes sent via SQL*Net to client
        385 bytes received via SQL*Net from client
          2 SQL*Net roundtrips to/from client
          1 sorts (memory)
          0 sorts (disk)
         11 rows processed
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 ;
11 rows selected.
Execution Plan
Plan hash value: 882605040
| Id | Operation                | Name | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT         |      | 1200 | 62400 |    80   (2)| 00:00:01 |
|* 1 | VIEW                    |      | 1200 | 62400 |    80   (2)| 00:00:01 |
|* 2 |   COUNT STOPKEY          |      |       |       |            |          |
|   3 |    VIEW                  |      | 40617 | 1546K|    80   (2)| 00:00:01 |
|* 4 |     SORT ORDER BY STOPKEY|      | 40617 | 2062K|    80   (2)| 00:00:01 |
|   5 |      TABLE ACCESS FULL   | T    | 40617 | 2062K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - filter("RN">=1190)
   2 - filter(ROWNUM<=1200)
   4 - filter(ROWNUM<=1200)
Statistics
          0 recursive calls
          0 db block gets
        343 consistent gets
          0 physical reads
          0 redo size
       1063 bytes sent via SQL*Net to client
        385 bytes received via SQL*Net from client
          2 SQL*Net roundtrips to/from client
          1 sorts (memory)
          0 sorts (disk)
         11 rows processed
SQL> select object_id, object_name, last_ddl_time from t, (select rid, rownum rn from (select rowid rid from t order by last_ddl_time desc) where rownum <= 1200) t1 where rn >= 1190 and t.rowid = t1.rid ;
11 rows selected.
Execution Plan
Plan hash value: 168880862
| Id | Operation                      | Name | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT               |       | 1200 | 91200 |   179   (2)| 00:00:03 |
|* 1 | HASH JOIN                     |       | 1200 | 91200 |   179   (2)| 00:00:03 |
|* 2 |   VIEW                         |       | 1200 | 30000 |    98   (0)| 00:00:02 |
|* 3 |    COUNT STOPKEY               |       |       |       |            |          |
|   4 |     VIEW                       |       | 40617 |   475K|    98   (0)| 00:00:02 |
|   5 |      INDEX FULL SCAN DESCENDING| T_IDX | 40617 |   793K|    98   (0)| 00:00:02 |
|   6 |   TABLE ACCESS FULL            | T     | 40617 | 2022K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - access("T".ROWID="T1"."RID")
   2 - filter("RN">=1190)
   3 - filter(ROWNUM<=1200)
Statistics
          0 recursive calls
          0 db block gets
        349 consistent gets
          0 physical reads
          0 redo size
       1063 bytes sent via SQL*Net to client
        385 bytes received via SQL*Net from client
          2 SQL*Net roundtrips to/from client
          0 sorts (memory)
          0 sorts (disk)
         11 rows processed
SQL> select object_id, object_name, last_ddl_time from (select t1.*, rownum rn from (select * from t order by last_ddl_time desc) t1 where rownum <= 1200) where rn >= 1190 order by last_ddl_time desc ;
11 rows selected.
Execution Plan
Plan hash value: 882605040
| Id | Operation           | Name | Rows     | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT      |     | 1200 | 62400 |    80   (2)| 00:00:01 |
|* 1 | VIEW                |     | 1200 | 62400 |    80   (2)| 00:00:01 |
|* 2 |   COUNT STOPKEY       |     |     |     |          |          |
|   3 |    VIEW            |     | 40617 | 1546K|    80   (2)| 00:00:01 |
|* 4 |     SORT ORDER BY STOPKEY|     | 40617 | 2062K|    80   (2)| 00:00:01 |
|   5 |      TABLE ACCESS FULL      | T     | 40617 | 2062K|    80   (2)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - filter("RN">=1190)
   2 - filter(ROWNUM<=1200)
   4 - filter(ROWNUM<=1200)
Statistics
     175 recursive calls
       0 db block gets
     388 consistent gets
       0 physical reads
       0 redo size
       1063 bytes sent via SQL*Net to client
     385 bytes received via SQL*Net from client
       2 SQL*Net roundtrips to/from client
       4 sorts (memory)
       0 sorts (disk)
      11 rows processed
SQL> set autotrace off
SQL> spool offAs you will see, the join query here has to have an ORDER BY clause at the end to ensure that records are correctly sorted. You can not rely on optimizer choosing NESTED LOOP join method and, as above example shows, when optimizer chooses HASH JOIN, oracle is free to return rows in no particular order.
The query that does not involve join always returns rows in the desired order. Adding an ORDER BY does add a step in the plan for the query using join but does not affect the other query.

Select All in large table

My product has some very large tables (using RANGE_PAGING). I want to allow the end user to do a Select All operation, and then press a command button to act on all selected rows, but want my backing code to detect a Select All has been done, rather than attempt to retrieve all rows from the table. Is this doable with a RichTable?
I'm porting an existing UI over to ADF -- the old UI handled this by having a separate select all button (column header), which would get unset if any row got unselected, and the backing code could interrogate that. I was wondering if there was a more ADF-ish way of handling this.
Using Oracle JDeveloper 11g Release 1 (11.1.1.6.0)
Edited by: user12614476 on Dec 5, 2011 1:58 PM
(added JDeveloper version info)

Are you really using 11.1.1.6? If so, I guess you should be asking in one of the internal Oracle forums.
But, no, there's no "more ADF-ish" way to my knowledge :)
John

How to subdivide 1 large TABLE based on the output of a VIEW

I am searching for a decent method / example code to subdivide a large table (into a global temp table (GTT) for further processing) based on a list of numeric/alphanumeric which is the resultset from a view.
I am groping with the following strategy in PL/SQL:
1 -- set up cursor, execute the view (so I have the list of identifiers)
2 -- create a second cursor (or loop?) which:
accepts each of the identifiers in turn
executes a query (EXECUTE IMMEDIATE?) on the larger table
INSERTs (or appends?) each resultset into the GTT
3 -- Then the GTT contains just the requires subset of the larger table for further processing and eventual import into iReport for reporting.
Can anyone point me to code that would "spoon feed" me on this? Or suggest the best / better way to go about it?
The scale of the issue here -- GTT is defined and ready to go, the larger table contains approx 40,000 rows and I need to extract a dozen subsets or so which add up to approx 1000 rows.
Thanks,
Rob

Welcome to the forum!
>
I am searching for a decent method / example code to subdivide a large table (into a global temp table (GTT) for further processing) based on a list of numeric/alphanumeric which is the resultset from a view.
Can anyone point me to code that would "spoon feed" me on this? Or suggest the best / better way to go about it?
The scale of the issue here -- GTT is defined and ready to go, the larger table contains approx 40,000 rows and I need to extract a dozen subsets or so which add up to approx 1000 rows.
>
No - there is no code to point you to.
As many of the previous responses indicate part of the concern is that you seem to have already chosen and partially implemented a solution but the information you provided makes us question whether you have adequately analyzed and defined the actual problem and processing that needs to happen. Here's why I have questions about your approach
1. GTT - a red flag issue - these tables are generally not needed in Oracle. So when you, or anyone says they plan to use one it raises a red flag. People want to be sure you really need one rather than not using a table at all, or just using a regular table instead of a GTT.
2. Double nested CURSOR loops - a DOUBLE red flag issue - this is almost always SLOW-BY-SLOW (row-by-row) processing at its worst. It is seldom needed, doesn't perform well and won't scale. People are going to question this choice and rightfully so.
3. EXECUTE IMMEDIATE - a red flag issue or at least a yellow/warning flag. This is definitely a legitimate methodology when it is needed but may times developers resort to it when it isn't needed because it seems easier than doing the hard work of actually defining ALL of the requirements. It seems easier because it appears that it will allow and work for those 'unexpected' things that seem to come up in new development.
Unfortunately most of those unexpected things come up because the developer did not adequately define all of the requirements. The code may execute when those things arise but it likely won't do the right thing.
Seeing all three of those red flag issues in the same question is like waving a red flag at a charging bull. The responses you get are all likely to be of the 'DO NOT DO THAT' variety.
You are correct that a work table is appropriate when there is business logic to be applied to a set of data that cannot be applied using SQL alone. Use a regular table unless
1. you plan to have multiple sessions working with the table simutaneously,
2. each session needs to work with ONLY their own data in that table and not data from other sessions
3. the data does NOT need to be available after the session ends
4. you actually need a GTT to take advantage of the automatic data preservation (ON COMMIT PRESERVE/DELETE) functionality
Remember - when a session ends the data in the GTT is gone. That can makek it very difficult to troubleshoot data related problems since a different session can't see what data is in the table. Even if a GTT is needed for the final product it is very useful to use a regular table so that the data can be examined after test runs to help find and fix problems. Then after development is complete and initial testing is done a GTT would be substituted and final testing performed.
So the main remaining question is why you need to perform multiple dynamic queries to get the data populated into the work table? Especially why is a nested cursor loop needed? My suspicion is that you have the queries stored in a query table and one of your loops extracts the query and executes it dynamically.
How many queries are we talking about? Do these queries change from run to run? Please provide more detail of the process and an example query for the selection filtering as well as a typical dynamic query you plan to use.

OutOfMemory error when trying to display large tables

We use JDeveloper 10.1.3. Our project uses ADF Faces + EJB3 Session Facade + TopLink.
We have a large table (over 100K rows) which we try to show to the user via an ADF Read-only Table. We build the page by dragging the facade findAllXXX method's result onto the page and choosing "ADF Read-only Table".
The problem is that during execution we get an OutOfMemory error. The Facade method attempts to extract the whole result set and to transfer it to a List. But the result set is simply too large. There's not enough memory.
Initially, I was under the impression that the table iterator would be running queries that automatically fetch just a chunk of the db table data at a time. Sadly, this is not the case. Apparently, all the data gets fetched. And then the iterator simply iterates through a List in memory. This is not what we needed.
So, I'd like to ask: is there a way for us to show a very large database table inside an ADF Table? And when the user clicks on "Next", to have the iterator automatically execute queries against the database and fetch the next chunk of data, if necessary?
If that is not possible with ADF components, it looks like we'll have to either write our own component or simply use the old code that we have which supports paging for huge tables by simply running new queries whenever necessary. Alternatively, each time the user clicks on "Next" or "Previous", we might have to intercept the event and manually send range information to a facade method which would then fetch the appropriate data from the database. I don't know how easy or difficult that would be to implement.
Naturally, I'd prefer to have that functionality available in ADF Faces. I hope there's a way to do this. But I'm still a novice and I would appreciate any advice.

Hi Shay,
We do use search pages and we do give the users the opportunity to specify search criteria.
The trouble comes when the search criteria are not specific enough and the result set is huge. Transferring the whole result set into memory will be disastrous, especially for servers used by hundreds of users simultaneously. So, we'll have to limit the number of rows fetched at a time. We should do this either by setting the Maximum Rows option for the TopLink query (or using rownum<=XXX inside the SQL), or through using a data provider that supports paging.
I don't like the first approach very much because I don't have a good recipe for calculating the optimum number of Maximum Rows for each query. By specifying some average number of, say, 500 rows, I risk fetching too many rows at once and I also risk filling the TopLink cache with objects that are not necessary. I can use methods like query.dontMaintainCache() but in my case this is a workaround, not a solution.
I would prefer fetching relatively small chunks of data at a time and not limiting the user to a certain number of maximum rows. Furthermore, this way I won't fetch large amounts of data at the very beginning and I won't be forced to turn off the caching for the query.
Regarding the "ADF Developer's Guide", I read there that "To create a table using a data control, you must bind to a method on the data control that returns a collection. JDeveloper allows you to do this declaratively by dragging and dropping a collection from the Data Control Palette."
So, it looks like I'll have to implement a collection which, in turn, implements the paging functionality that I need. Is the TopLink object you are referring to some type of collection? I know that I can specify a collection class that TopLink should use for queries through the query.useCollectionClass(...) method. But if TopLink doesn't provide the collection I need, I will have to write that collection myself. I still haven't found the section in the TopLink documentation that says what types of Collections are natively provided by TopLink. I can see other collections like oracle.toplink.indirection.IndirectList, for example. But I have not found a specific discussion on large result sets with the exception of Streams and Cursors and I feel uneasy about maintaining cursors between client requests.
And I completely agree with you about reading the docs first and doing the programming afterwards. Whenever time permits, I always do that. I have already read the "ADF Developer's Guide" with the exception of chapters 20 and 21. And I switched to the "TopLink Developer's Guide" because it seems that we must focus on the model. Unfortunately, because of the circumstances, I've spent a lot of time reading and not enough time practicing what I read. So, my knowledge is kind of shaky at the moment and perhaps I'm not seeing things that are obvious to you. That's why I tried using this forum -- to ask the experts for advice on the best method for implementing paging. And I'm thankful to everyone who replied to my post so far.

Using dbms_metadata to get defination of large table

Hi,
I have very very large table definition table with partitions and sub-partitions and I want to create that table in some other db.
I am trying below spooled query but its not retrieving complete definition of the table, any other setting I should follow
spool t.sql
set pagesize 0
set long 2000000
select dbms_metadata.get_ddl
I am using oracle 10g r2
Thanks,
Pankaj

Thanks you all for your replied, following are observation after settings suggested
With setting
==================
set longchunksize 2000000
set long 2000000
set lines 2000
set pages 0
-sh-3.00$ cat t.sql | wc
10819 23419 21635001
=================
With setting
set long 100000
set pages 0
-sh-3.00$ cat t.sql | wc
3452 7326 279269
Result => None worked for me :(

Slow query due to large table and full table scan

Hi,
We have a large Oracle database, v 10g. Two of the tables in the database have over one million rows.
We have a few queries which take a lot of time to execute. Not always though, it that seems when load is high the queries tend
to take much longer. Average time may be 1 or 2 seconds, but maxtime can be up to 2 minutes.
We have now used Oracle Grid to help us examine the queries. We have found that some of the queries require two or three full table scans.
Two of the full table scans are of the two large tables mentioned above.
This is an example query:
SELECT table1.column, table2.column, table3.column
FROM table1
JOIN table2 on table1.table2Id = table2.id
LEFT JOIN table3 on table2.table3id = table3.id
WHERE table1.id IN(
SELECT id
FROM (
(SELECT a.*, rownum rnum FROM(
SELECT table1.id
FROM table1,
table2,
table3
WHERE
table1.table2id = table2.id
AND
table2.table3id IS NULL OR table2.table3id = :table3IdParameter
) a
WHERE rownum <= :end))
WHERE rnum >= :start
Table1 and table2 are the large tables in this example. This query starts two full table scans on those tables.
Can we avoid this? We have, what we think are, the correct indexes.
/best regards, Håkan

>
Hi Håkan - welcome to the forum.
We have a large Oracle database, v 10g. Two of the tables in the database have over one million rows.
We have a few queries which take a lot of time to execute. Not always though, it that seems when load is high the queries tend
to take much longer. Average time may be 1 or 2 seconds, but maxtime can be up to 2 minutes.
We have now used Oracle Grid to help us examine the queries. We have found that some of the queries require two or three full table scans.
Two of the full table scans are of the two large tables mentioned above.
This is an example query:Firstly, please read the forum FAQ - top right of page.
Please format your SQL using tags [code /code].
In order to help us to help you.
Please post table structures - relevant (i.e. joined, FK, PK fields only) in the form - note use of code tags - we can just run table create script.
CREATE TABLE table1
Field1 Type1,
Field2 Type2,
FieldN TypeN
);Then give us some table data - not 100's of records - just enough in the form
INSERT INTO Table1 VALUES(Field1, Field2.... FieldN);
..Please post EXPLAIN PLAN - again with tags.
HTH,
Paul...
/best regards, Håkan

Updating Large Tables

Hi,
I was asked the following during an interview
U have a large table with millions rows and want to add a column. What is the best way to do it without effecting the preformance of DB
Also u have a large table with million rows how do u organise the indexes
My answer was to coalasce the indexes
I was wondering what is the best answers for these questions
Thanks

Adding a column to a table, even a really big one is trivial and will have no impact on the performance of the database. Just do:
ALTER TABLE t ADD (new_column DATATYPE);This is simply an update to the data dictionary. Aside from the few milliseconds that Oracle will lock some dicitonary tables (no different than the locks held if you update a column in a user table) there will be no impact on the performance of the database. Now, populatng that column would be a different kettle of fish, and would depend on how (i.e. single value for all rows, calculated based on other columns) the column needs to be populated.
I would have asked for clarification on what they meant by "oraganise the indexes". If they meant what tablespaces should they go in, I would say in the same tablespace as other objects of similar size (You are using locally managed tablespaces aren't you?). If they meant what indexes would you create, I would say that I would create the indexes neccessary to answer the queries that you run.
HTH
John

Joining on two large tables give breaks connections?

I am doing inner join on two large tables (172,818 and 146,215) give breaks connections. Using Oracle 8.1.7.0.0
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Oraigannly I was trying to do alter table to add constrints, it gave the error aswell.
ALTER TABLE a ADD (CONSTRAINT a_FK
FOREIGN KEY (a_ID, a_VERSION)
REFERENCES b(b_ID, b_VERSION)
DEFERRABLE INITIALLY IMMEDIATE)
Also gives same error. Trace file does not make sense to me.

Thanks for the reply, no luck yet.
SQL> show parameter optimizer_max_permutations ;
NAME TYPE VALUE
optimizer_max_permutations integer 80000
SQL> show parameter resource_limit ;
NAME TYPE VALUE
resource_limit boolean FALSE
SQL>

How to instantiate multiple large tables while DB is online

Hello Experts,
I've set up goldengate on a RHEL 11.2.0.3 Cluster to a HP Unix Itanium 10.2.0.4 database.
I've got a sample table replicated successfully, and changes are moving over to the target system.
However, now comes the real thing, I have about 100 + varying size tables, one 1TB, then a few 30 GB tables, and then finally a bunch of smaller tables.
I've not been able to find a clean way to do this while the db is online.
The afterscn is good, but it works for the replicat not at the table level...
Any help is appreciated.
Best Regards.

Since I've received no updates here, I'd put in an SR with Oracle Support.
The suggestion was to add additional large tables online using another new replicat, with the afterscn option for the replicat.
Once the changes are all synced up and the replication has been running for a while, stop the Pump on the source, stop the new replicat, delete it, and move the table configuration to an existing replicat, and startup the stopped pump, and you're done with the online re-instantiation.
Am putting it to the test, but logically it sounds feasible.

Using workspaces with large tables

Hello
I've got a few large tables (6-10GB+) that will have around 500k new rows added on a daily basis as part of an overnight batch job. No rows are ever updated, only inserted or deleted and then re-inserted. I want to stop the process that adds the new rows from being an overnight batch to being a near real time process i.e. a queue will be populated with requests to rebuild the content of these tables for specific parent ids, and a process will consume those requests throughout the day rather than going through the whole list in one go.
I need to provide views of the data asof a point in time i.e. what was the content of the tables at close of business yesterday, and for this I am considering using workspaces.
I need to keep at least 10 days worth of data and I was planning to partition the table and drop one partition every day. If I use workspaces, I can see that oracle creates a view in place of the original table and creates a versioned table with the LT suffix - this is the table name returned by DBMSMW.GetPhysicalTableName. Would it be considered bad practice to drop partitions from this physical table as I would do with a non version enabled table? If so, what would be the best method for dropping off old data?
Thanks in advance
David

I've just spotted the workspace manager forum, I'll post there. :-)

HS connection to MySQL fails for large table

Hello,
I have set up an HS to a MySql 3.51 dabatabe using an ODBC DNS. My Oracle box has version 10.2.0.1 running in Windows 2003 R2. MySQL version is 4.1.22 running on a different machine with the same OS.
I completed the connection through a database link, which works fine in SQLPLUS when selecting small MySQL Tables. However, I keep getting an out of memory error when selecting certain large table from the MySQL database. Previously, I had tested the DNS and ran the same SELECT in Access and it doesn't give any error. This is the error thrown by SQLPLUS:
SQL> select * from progressnotes@mysql_rmg where "encounterID" = 224720;
select * from progressnotes@mysql_rmg where "encounterID" = 224720
ERROR at line 1:
ORA-00942: table or view does not exist
[Generic Connectivity Using ODBC][MySQL][ODBC 3.51
Driver][mysqld-4.1.22-community-nt]Lost connection to MySQL server during query
(SQL State: S1T00; SQL Code: 2013)
ORA-02063: preceding 2 lines from MYSQL_RMG
I traced the HS connection and here is the result from the .trc file:
Oracle Corporation --- THURSDAY JUN 12 2008 11:19:51.809
Heterogeneous Agent Release
10.2.0.1.0
(0) [Generic Connectivity Using ODBC] version: 4.6.1.0.0070
(0) connect string is: defTdpName=MYSQL_RMG;SYNTAX=(ORACLE8_HOA, BASED_ON=ORACLE8,
(0) IDENTIFIER_QUOTE_CHAR="",
(0) CASE_SENSITIVE=CASE_SENSITIVE_QUOTE);BINDING=<navobj><binding><datasources><da-
(0) tasource name='MYSQL_RMG' type='ODBC'
(0) connect='MYSQL_RMG'><driverProperties/></datasource></datasources><remoteMachi-
(0) nes/><environment><optimizer noFlattener='true'/><misc year2000Policy='-1'
(0) consumerApi='1' sessionBehavior='4'/><queryProcessor parserDepth='2000'
(0) tokenSize='1000' noInsertParameterization='true'
noThreadedReadAhead='true'
(0) noCommandReuse='true'/></environment></binding></navobj>
(0) ORACLE GENERIC GATEWAY Log File Started at 2008-06-12T11:19:51
(0) hoadtab(26); Entered.
(0) Table 1 - PROGRESSNOTES
(0) [MySQL][ODBC 3.51 Driver][mysqld-4.1.22-community-nt]MySQL client ran out of
(0) memory (SQL State: S1T00; SQL Code: 2008)
(0) (Last message occurred 2 times)
(0)
(0) hoapars(15); Entered.
(0) Sql Text is:
(0) SELECT * FROM "PROGRESSNOTES"
(0) [MySQL][ODBC 3.51 Driver][mysqld-4.1.22-community-nt]Lost connection to MySQL
(0) server during query (SQL State: S1T00; SQL Code: 2013)
(0) (Last message occurred 2 times)
(0)
(0) [A00D] Failed to open table MYSQL_RMG:PROGRESSNOTES
(0)
(0) [MySQL][ODBC 3.51 Driver]MySQL server has gone away (SQL State: S1T00; SQL
(0) Code: 2006)
(0) (Last message occurred 2 times)
(0)
(0) [MySQL][ODBC 3.51 Driver]MySQL server has gone away (SQL State: S1T00; SQL
(0) Code: 2006)
(0) (Last message occurred 2 times)
(0)
(0) [S1000] [9013]General error in nvITrans_Commit - rc = -1. Please refer to the
(0) log file for details.
(0) [MySQL][ODBC 3.51 Driver]MySQL server has gone away (SQL State: S1T00; SQL
(0) Code: 2006)
(0) (Last message occurred 2 times)
(0)
(0) [S1000] [9013]General error in nvITrans_Rollback - rc = -1. Please refer to
(0) the log file for details.
(0) Closing log file at THU JUN 12 11:20:38 2008.
I have read the MySQL documentation and apparently there's a "Don't Cache Result (forward only cursors)" parameter in the ODBC DNS that needs to be checked in order to cache the results in the MySQL server side instead of the Driver side, but checking that parameter doesn't work for the HS connection. Instead, the SQLPLUS session throws the following message when selecting the same large table:
SQL> select * from progressnotes@mysql_rmg where "encounterID" = 224720;
select * from progressnotes@mysql_rmg where "encounterID" = 224720
ERROR at line 1:
ORA-02068: following severe error from MYSQL_RMG
ORA-28511: lost RPC connection to heterogeneous remote agent using
SID=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.120)(PORT=1521))(CONNECT_DATA=(SID=MYSQL_RMG)))
Curiously enough, after checking the parameter, the Access connection through the DNS ODBS seems to improve!
Is there an aditional parameter that needs to be set up in the inithsodbc.ora perhaps? These are current HS paramters:
# HS init parameters
HS_FDS_CONNECT_INFO = MYSQL_RMG
HS_FDS_TRACE_LEVEL = ON
My SID_LIST_LISTENER entry is:
(SID_DESC =
(PROGRAM = HSODBC)
(SID_NAME = MYSQL_RMG)
(ORACLE_HOME = D:\oracle\product\10.2.0\db_1)
Finally, here is my TNSNAMES.ORA entry for the HS connection:
MYSQL_RMG =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.0.0.120)(PORT = 1521))
(CONNECT_DATA =
(SID = MYSQL_RMG)
(HS = OK)
Your advice will be greatly appeciated,
Thanks,
Luis
Message was edited by:
lmconsite

First of all please be aware HSODBC V10 has been desupported and DG4ODBC should be used instead.
The root cause the problem you describe could be related to a timeout of the ODBC driver (especially while taking care of the comment: it happens only for larger tables):
(0) [MySQL][ODBC 3.51 Driver]MySQL server has gone away (SQL State: S1T00; SQL
(0) Code: 2006)
indicates the Driver or the DB abends the connection due to a timeout.
Check out the wait_timeout mysql variable on the server and increase it.

Partitioning large table

I'm in need of some expert advice on how best to partition the following table in Oracle 9i. It's expected to hold billions of records. This table is expected to receive 10,000,000 new records every hour or so.
I've tried a composite range-hash partitioning scheme, however it does not appear to be improving the speed of INSERTs. I've created the tablespaces named "part0" through "part9", which all happen to be on the same disk. I realize that optimally these would be on separate disks, however I expected to see a larger improvement over the same table without partitions which doesn't appear to be happening. Here's it's current definition:
CREATE TABLE sp_device_usage
(id NUMBER(15) NOT NULL,
device_id NUMBER(9) NOT NULL,
datastream_id NUMBER(9) NOT NULL,
timestamp NUMBER(10) NOT NULL,
usage_counter NUMBER(15) NOT NULL,
CONSTRAINT sp_device_usage_pk PRIMARY KEY (id) USING INDEX TABLESPACE cni_index,
CONSTRAINT sp_device_usage_fk1 FOREIGN KEY (device_id) REFERENCES encore_device,
CONSTRAINT sp_device_usage_fk2 FOREIGN KEY (datastream_id) REFERENCES sp_datastream
PARTITION BY RANGE (timestamp)
SUBPARTITION BY HASH (device_id) SUBPARTITIONS 10 (
PARTITION sp_device_usage_ptn1 VALUES LESS THAN (100000) TABLESPACE part0,
PARTITION sp_device_usage_ptn2 VALUES LESS THAN (200000) TABLESPACE part1,
PARTITION sp_device_usage_ptn3 VALUES LESS THAN (300000) TABLESPACE part2,
PARTITION sp_device_usage_ptn4 VALUES LESS THAN (400000) TABLESPACE part3,
PARTITION sp_device_usage_ptn5 VALUES LESS THAN (500000) TABLESPACE part4,
PARTITION sp_device_usage_ptn6 VALUES LESS THAN (600000) TABLESPACE part5,
PARTITION sp_device_usage_ptn7 VALUES LESS THAN (700000) TABLESPACE part6,
PARTITION sp_device_usage_ptn8 VALUES LESS THAN (800000) TABLESPACE part7,
PARTITION sp_device_usage_ptn9 VALUES LESS THAN (900000) TABLESPACE part8,
PARTITION sp_device_usage_ptn10 VALUES LESS THAN (maxvalue) TABLESPACE part9
CREATE INDEX sp_device_usage_idx1 ON
sp_device_usage(device_id) TABLESPACE cni_index
CREATE INDEX sp_device_usage_idx2 ON
sp_device_usage(datastream_id) TABLESPACE cni_index
CREATE SEQUENCE sp_device_usage_seq START WITH 1000 INCREMENT BY 1 MINVALUE 1 NOCACHE NOCYCLE NOORDER;

INSERT operations into a partitioned table won't necessarily be any faster than INSERT operations into a non-partitioned table. For partitioned tables, Oracle has to determine which partition to insert each record into, which is an extra step in the partitioned universe. Since you will, for the most part, be inserting into the same partition, you're basically doing the normal table insert with additional partition key checking overhead. Partitioning is designed to speed the retrieval of and, to a lesser extent, update operations on data as well as to improve managability.
Justin Cave <[email protected]>
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC

LARGE TABLE IN ORACLE

Similar Messages

Maybe you are looking for