Select 10000 rows from Oracle in shortest way
Looking for a good desicions for trasfering more than 10000 rows from
oracle to mysql using only jdbc. Connect is mostly slow about 1 row in
a second, rights on oracle are just for select, update, insert
operations. I think if I can devide 10000 rows in 10 parts and transfer
them in concurent threads and connections. Any suggestion can help me
to solve this problem.
See java.sql.Statement.setFetchSize(), e.g.I'm not sure that setFetchSize will have any impact on your performance. It is mostly a suggestion to the JDBC driver, and while in theory the suggestion may help, in practice I have not seen any signficant benefit when using Oracle.
To test the speed of your fetch, try issuing your select, then loop through the resultset, getting all the columns, just don't do anything with them. You could try changing the fetchSize to see if it does have an impact in your specific architecture.
Now that you know how fast you can 'get' all the rows, how much different is it then the 1 second per row that you are seeing when selecting and inserting (I'm assuming you haven't done this yet, sorry if you already have)?
Assuming that you can select all rows in less then the 1 second per row in your original test; there are several things you can do to increase the speed of inserts. Try using a PreparedStatement if you are not already using one . Try using the .addBatch() and .executeBatch(). You can try changing the count of rows that are inserted on each executeBatch() command.
If all your inserts are inserting into a single table, then I don't think multiple threads will help you, and may in fact hurt you because you could run into locking issues in the mySQL database. If you are inserting into multiple tables, then it is possible that multi-threading, done correctly, may provide some increase in overall speed.
Oracle can be accessed with 3 different JDBC drivers. If you are using Oracle 8i or 9i, then try using the OCI8 driver instead of the thin driver. The OCI8 driver may provide some performance benefits when doing mass inserts in the older versions of Oracle. Do not use the JDBC-ODBC bridge as that will provide the worst performance.
If I was going to move data between two different databases, and the vendor did not provide a utility to do that specifically then I would try and use the 1st db vendors unload utility to unload into a text file, and the 2nd db vendors load utility to load from that text file. These utilities have been optimized for speed far and above anything that will ever be available to you as a Java programmer using JDBC. That isn't always possible based on the architecture of your solution, but it is always preferrable.
Best of luck to you.
Similar Messages
-
How to efficiently select random rows from a large table ?
Hello,
The following code will select 5 rows out of a random set of rows from the emp (employee) table
select *
from (
select ename, job
from emp
order by dbms_random.value()
where rownum <= 5my concern is that the inner select will cause a table scan in order to assign a random value to each row. This code when used against a large table can be a performance problem.
Is there an efficient way of selecting random rows from a table without having to do a table scan ? (I am new to Oracle, therefore it is possible that I am missing a very simple way to perform this task.)
thank you for your help,
John.
Edited by: 440bx on Jul 10, 2010 6:18 PMHave a look at the SAMPLE clause of the select statement. The number in parenthesis is a percentage of the table.
SQL> create table t as select * from dba_objects;
Table created.
SQL> explain plan for select * from t sample (1);
Explained.
SQL> @xp
PLAN_TABLE_OUTPUT
Plan hash value: 2767392432
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 725 | 70325 | 289 (1)| 00:00:04 |
| 1 | TABLE ACCESS SAMPLE| T | 725 | 70325 | 289 (1)| 00:00:04 |
8 rows selected. -
How to select a row from duplicate set of records?
I want to select a row from a duplicate set of records.
below is the explanation of my requirement.
select * from test_dup;
COL_BILL COL_SERV COL_SYS
b1 s1 c
b1 s1 g
b1 s2 c
b1 s2 g
b2 s2 g
b2 s3 c
b2 s3 g
b3 s3 c
Here what I want is, for a distinct col_sys if col_bill and col_serv is same then I need the row where col_sys='c'
from the above result set I need the following:
b1 s1 c
b1 s2 c
b2 s3 c
I am using the following SQL query which is giving me the correct result set. But it will hamper the performance because there are total 45 columns in the table and total volume is around 50 million.
select * from test_dup where col_bill||col_serv in (
select col_bill||col_serv from (
select col_bill,col_serv,count(*) from test_dup
where col_sys in ('c','g')
group by col_bill,col_serv having count(*) >1)) and
col_sys='c';
Can anyone please provide me the optimize SQL query for the same.Hi,
Another way,
SQL> with test_dup
as
select 'b1' col_bill, 's1' col_serv, 'c' col_sys from dual union all
select 'b1', 's1', 'g' from dual union all
select 'b1', 's2', 'c' from dual union all
select 'b1', 's2', 'g' from dual union all
select 'b2', 's2', 'g' from dual union all
select 'b2', 's3', 'c' from dual union all
select 'b2', 's3', 'g' from dual union all
select 'b3', 's3', 'c' from dual
select col_bill, col_serv, min(col_sys) col_sys
from test_dup
where col_sys in ('c', 'g')
group by col_bill, col_serv
having count( * ) > 1
and count(nullif(col_sys, 'g')) > 0;
CO CO C
b1 s1 c
b2 s3 c
b1 s2 c
3 rows selected.Regards
Peter
Edited by: Peter on Feb 18, 2009 1:10 AM
- Added test data, thanks Karthick -
Display distinct rows from Oracle table without using "DISTINCT" keyword.
How to retrieve distinct rows from oracle table without using 'DISTINCT' keyword in SQL?
Thanks in advance.
MihirWelcome to the forum.
Besides GROUP BY you can use UNIQUE instead of DISTINCT as well, but that's probably not wanted here ;) , and the ROW_NUMBER() analytic:
SQL> create table t as
2 select 1 col1 from dual union all
3 select 1 from dual union all
4 select 2 from dual union all
5 select 3 from dual union all
6 select 4 from dual union all
7 select 4 from dual;
Table created
SQL> select col1 from t;
COL1
1
1
2
3
4
4
6 rows selected
SQL> select distinct col1 from t;
COL1
1
2
3
4
SQL> select unique col1 from t;
COL1
1
2
3
4
SQL> select col1 from t group by col1;
COL1
1
2
3
4
SQL> select col1
2 from ( select col1
3 , row_number() over (partition by col1 order by col1) rn
4 from t
5 )
6 where rn=1;
COL1
1
2
3
4 -
How to select multiple rows from List Of Values
Hello,
I use ADF 11g to create my list of values (LOV). I want to select multiple rows from it. but i can't.
so how i can select many rows to set them in my adf table.
Thank in advanceHi,
LOV is map to an attribute in the viewObject so it will return only one value or more values from selected row. You can't select multiple rows from LOV.
But you can do this by using popup which you can select multiple rows and insert the selected rows to another table.
This blog post explain how to achieve this :
http://husaindalal.blogspot.com/2009/11/search-from-popup-and-add-to-new-table.html#comments
Sameh Nassar -
How can i select some row from multiple row in the same group of data
I want to select some row from multiple row in the same group of data.
ColumnA
Column B
1 OK
1 NG
2 NG
2 NG
3 OK
3 OK
I want the row of group of
ColumnA if ColumnB contain even 'NG'
row , select only one row which Column B = 'NG'
the result i want =
ColumnA Column B
1 NG
2 NG
3 OK
Thank youThat's some awful explanation, but I think this is what you were driving at:
DECLARE @forumTable TABLE (a INT, b CHAR(2))
INSERT INTO @forumTable (a, b)
VALUES
(1, 'OK'),(1, 'NG'),
(2, 'NG'),(2, 'NG'),
(3, 'OK'),(3, 'OK')
SELECT f.a, MIN(COALESCE(f2.b,f.b)) AS b
FROM @forumTable f
LEFT OUTER JOIN @forumTable f2
ON f.a = f2.a
AND f.b <> f2.b
GROUP BY f.a -
When selecting a row from a view with a nested table I want just ONE entry returned
Does a nested table in a view "EXPLODE" all values ALWAYS no matter the where clause for the nested table?
I want to select ONE row from a view that has columns defined as TYPE which are PL/SQL TABLES OF other tables.
when I specify a WHERE clause for my query it gives me the column "EXPLODED" with the values that mathc my WHERE clause at the end of the select.
I dont want the "EXPLODED" nested table to show just the entry that matches my WHERE clause. Here is some more info:
My select statement:
SQL> select * from si_a31_per_vw v, TABLE(v.current_allergies) a where a.alg_seq
=75;
AAAHQPAAMAAAAfxAAA N00000 771 223774444 20 GREGG
CADILLAC 12-MAY-69 M R3
NON DENOMINATIONAL N STAFF USMC N
U
E06 11-JUN-02 H N
05-JAN-00 Y Y
USS SPAWAR
353535 USS SPAWAR
SI_ADDRESS_TYPE(NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NUL
L, NULL)
SI_ADDRESS_TAB()
SI_ALLERGY_TAB(SI_ALLERGY_TYPE(69, 'PENICILLIN', '11-JUN-02', NULL), SI_ALLERGY
TYPE(74, 'SHELLFISH', '12-JUN-02', NULL), SIALLERGY_TYPE(68, 'PEANUTS', '13-J
UN-02', NULL), SI_ALLERGY_TYPE(75, 'STRAWBERRIES', '13-JUN-02', NULL))
SI_ALLERGY_TAB()
75 STRAWBERRIES 13-JUN-02
*******Notice the allergy entry of 75, Strawberries, 13-JUN-02 at the
end. This is what I want not all the other exploded data.
SQL> desc si_a31_per_vw
Name Null? Type
........ Omitted uneeded previous column desc because of metalink
character limit but the view is bigger then this.......
DEPT_NAME VARCHAR2(20)
DIV_NAME VARCHAR2(20)
ADDRESSES SI_ADDRESS_TAB
CURRENT_ALLERGIES SI_ALLERGY_TAB
DELETED_ALLERGIES SI_ALLERGY_TAB
SQL> desc si_allergy_tab
si_allergy_tab TABLE OF SI_ALLERGY_TYPE
Name Null? Type
ALG_SEQ NUMBER
ALG_NAME VARCHAR2(50)
START_DATE DATE
STOP_DATE DATE
SQL> desc si_allergy_type
Name Null? Type
ALG_SEQ NUMBER
ALG_NAME VARCHAR2(50)
START_DATE DATE
STOP_DATE DATECan you explain what do you mean by the following?
"PL/SQL tables (a.k.a. Index-by tables) cannot be used as the basis for columns and/or attributes"There are three kinds of collections:
(NTB) Nested Tables
(VAR) Varrying Arrays
(IBT) Index-by Tables (the collection formerly known as "PL/SQL tables")
NTB (and VAR) can be defined as persistent user defined data types, and can be used in table DDL (columns) and other user defined type specifications (attributes).
SQL> CREATE TYPE my_ntb AS TABLE OF INTEGER;
SQL> CREATE TABLE my_table ( id INTEGER PRIMARY KEY, ints my_ntb );
SQL> CREATE TYPE my_object AS OBJECT ( id INTEGER, ints my_ntb );
/IBT are declared inside stored procedures only and have slightly different syntax from NTB. Only variables in stored procedures can be based on IBT declarations.
CREATE PROCEDURE my_proc IS
TYPE my_ibt IS TABLE OF INTEGER INDEX BY BINARY_INTEGER; -- now you see why they are called Index-by Tables
my_ibt_var my_ibt;
BEGIN
NULL;
END;That sums up the significant differences as it relates to how they are declared and where they can be referenced.
How are they the same?
NTB and VAR can also be (non-persistently) declared in stored procedures like IBTs.
Why would you then ever use IBTs?
IBTs are significantly easier to work with, since you don't have to instantiate or extend them as you do with NTB and VAR, or
Many other highly valuable PL/SQL programs make use of them, so you have to keep your code integrated/consistent.
There's a lot more to be said, but I think this answers the question posed by Sri.
Michael -
Newbie question: Select one row from table in PL/SQL
Hi,
I want to select one row from the table Employee where Emplyoyee Number is say 200. This is a simple SQL query, but I don't know the equivalent PL/SQL format. I will have 3 out params here - Id itself, Name, Salary. I will then have to populate a java resultset object from these out params.
Later, I'll have to use cursors to retrieve more than one row.
Thanks for any help.Perhaps something like
CREATE OR REPLACE PROCEDURE get_employee( l_id IN OUT employee.id%TYPE,
l_name OUT employee.name%TYPE,
l_salary OUT employee.salary%TYPE )
AS
BEGIN
SELECT name, salary
INTO l_name, l_salary
FROM employee
WHERE id = l_id;
END;Justin
Distributed Database Consulting, Inc.
http://www.ddbcinc.com/askDDBC -
Select a row from a table and throw an url
Hi experts,
I'm trying to select a row from a table in Visual Composer. What I need to do it's to click over this row and throw an Url. At the moment, I've added a column with a Pushbutton with the formula to show the link; it works fine. But I need to do it without buttons just selecting the row.
Is there anyboby who knows about that?
Thanks a lot.
BelenHi
To do this you will have to use a data store and a guard condition on the line which is triggered by the select action. You need to store a value in the data store each time a select action is performed. The guard condition should check for this and only perform the action when the value in the data store is greater than 0.
Jarrod Williams -
Select multiple rows from dual table
Is it possible to select multiple rows from dual table using a single select statement.
i.e., i want the out put to be
column_name
1
2
3
4
Edited by: vidya.ramachandra on Dec 14, 2009 8:24 AMAside from the fact you're responding to an old thread...
1002424 wrote:
While using CONNECT BY, I see it always leave behind one row.
Suppose I have a condition based on which I have to generate constant rows like
SELECT 1 FROM DUAL WHERE ROWNUM < N;
Here if N = 0, still it gives out single row.... you are obviously doing something wrong in your code elsewhere, because that SQL statement does not always return a single row...
SQL> SELECT 1 FROM DUAL WHERE ROWNUM < 0;
no rows selected
SQL> -
To select particular rows from order by select query
Hi all,
I want to select particular rows from a order by select query like...
select * from emp order by ename;
If i pass 3 and 7 dynamically in where clause or something like that, it has to show rows from 3 to 7 only.
Thanks in advance
PalThis?
SQL> select *
from (
select e.*,
row_number() over (order by ename) rn
from emp e)
where rn in (3,7)
EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO RN
7698 BLAKE MANAGER 7839 01-MAY-81 2850 30 3
7566 JONES MANAGER 7839 02-APR-81 2975 20 7
2 rows selected. -
Easy Question:Select many rows from a table and execute BAPI for these rows
Hi Experts,
I have created one WD project. The WD project fetches some records of backend using BAPI and displays in a table. I have to select some rows from the table and then execute BAPI for selected rows.
How I can select multiple records from the table and then execute another BAPI for selected rows.
Regards,
GaryHi,
In the Node which you binded to the table create one more attribute of type boolean.
For example your node is as below:
//Table Node
TableNode
> Att1
> Att2
> isSelected(boolean) - Newly created attribute for this requirement.
//Result Node contains the elements selected in TableNode
ResultNode
>Att1
>Att2
Now in the table create one more Column with Checkbox as tablecell editor. Now bind this boolean attribute to that check box.
Now in the code you can get the selected rows by user as below:
for(int i=0;i<TableNode().size();i++)
if(wdContext.nodeTableNode().getTableNodeElementAt(i).getIsSelected()==true)
IPrivateTestView.IResultNode element=wdContext.createResultNodeElement();
element.setAtt1(wdContext.nodeTableNode().getTableNodeElementAt(i).getAtt1());
element.setAtt2(wdContext.nodeTableNode().getTableNodeElementAt(i).getAtt2());
wdContext.nodeResultNode().addElement(element);
Regards,
Charan -
Selecting Multiple Rows from ALV GRID Display
Hi,
I am having a ALV GRID Display. I want to select multiple rows from the Output and move them to an internal table.
Please let me know how do I acheive this.
Thanks in advance,
Ishaq.Hi,
Have a look on the following code. It displays the selected rows which hv been selected in basic list.
TABLES:
spfli.
TYPE-POOLS:
slis.
DATA:
BEGIN OF t_spfli OCCURS 0,
checkbox.
INCLUDE STRUCTURE spfli.
DATA: END OF t_spfli.
DATA:
t_sspfli LIKE STANDARD TABLE OF t_spfli .
DATA:
fs_spfli LIKE LINE OF t_sspfli.
DATA:
fs_layout TYPE slis_layout_alv,
w_program TYPE sy-repid.
SELECT *
FROM spfli
INTO CORRESPONDING FIELDS OF TABLE t_spfli.
*fs_layout-info_fieldname = 'COLOR'.
fs_layout-box_fieldname = 'CHECKBOX'.
w_program = sy-repid.
CALL FUNCTION 'REUSE_ALV_LIST_DISPLAY'
EXPORTING
i_callback_program = w_program
i_callback_pf_status_set = 'FLIGHT'
i_callback_user_command = 'SPFLI_INFO'
i_structure_name = 'SPFLI'
is_layout = fs_layout
TABLES
t_outtab = t_spfli
EXCEPTIONS
program_error = 1
OTHERS = 2.
IF sy-subrc <> 0.
MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
*& Form FLIGHT
text
-->RT_EXTAB text
FORM flight USING rt_extab TYPE slis_t_extab..
SET PF-STATUS 'FLIGHT' EXCLUDING rt_extab.
ENDFORM. "FLIGHT
*& Form SPFLI_INFO
text
-->UCOMM text
-->SELFIELD text
FORM spfli_info USING ucomm LIKE sy-ucomm
selfield TYPE slis_selfield.
selfield-refresh = 'X'.
CASE ucomm.
WHEN 'FLIGHT'.
LOOP AT t_spfli.
IF t_spfli-checkbox = 'X'.
t_spfli-checkbox = ' '.
t_spfli-color = 'C51'.
MODIFY t_spfli TRANSPORTING checkbox.
fs_spfli = t_spfli.
APPEND fs_spfli TO t_sspfli.
ENDIF.
ENDLOOP.
WHEN 'EXIT'.
LEAVE PROGRAM.
ENDCASE.
CLEAR fs_spfli.
fs_layout-info_fieldname = 'COLOR'.
fs_layout-confirmation_prompt = 'X'.
fs_layout-key_hotspot = 'X'.
CALL FUNCTION 'REUSE_ALV_LIST_DISPLAY'
EXPORTING
i_callback_program = w_program
i_structure_name = 'SFLIGHT'
is_layout = fs_layout
TABLES
t_outtab = t_sspfli
EXCEPTIONS
program_error = 1
OTHERS = 2.
IF sy-subrc <> 0.
MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
REFRESH t_sspfli.
ENDFORM. "SPFLI_INFO
Regards,
Chandu -
Selecting multiple rows from List-component
Hi
Could someone give me an example how to programmatically select multiple rows from List-component?
I know that this selects one row: lst_example.selectedIndex = 1;
But how about selectin indexes 1,2 and 4 for example?selectedIndices
A Vector of ints representing the indices of the currently selected item or
items...
var si:Vector.<int> = new Vector.<int>;
si.push(1);
si.push(2);
si.push(4);
list.selectedIndices = si; -
Randomly selecting some rows from the database table
Hi can some one help me in selecting some rows from a database table which has around 90,000 rows.
Thanks.One thing you might try is the "sample" clause if you have 8i which is supposed to return a random percentage of the table. Say for example, you have a sequence number on your table as the pkey. Then you might try:
select * from <table_name> where pkey in(select pkey from <table_name> sample(10));
This should give you a random 10 percent of the rows in the table but I tried this once and the results seemed unpredictable. For example it returned a different number of rows each time even though the number of rows in the table didn't change.
Hope this works for you.
Maybe you are looking for
-
Please guys help me to fix my problem abovementioned.
-
How to create a Purchase Requisition for a material component in WBS or NW?
Hello All, I have created a project with the following structure: Project Defination (Construction of Sub station) Main WBS Level 1 WBS Level 2 WBS Network for level 2 W
-
REP-501: Unable to connect to the specified database
Hi, the following error occurs when running a paper report from forms:[ 2006/8/10 9:38:19:493] State 56016 (JobManager:updateJobStatus): Job 22 status is: Terminated with error: REP-501: Unable to connect to the specified database. [2006/8/10 9:38:19
-
Transfering music from my i tunes to m new i phone 5
I connected my new i phone 5 to my computer in order to transfer my music from i tunes, but my iphone does not show anywhere. Is it possible that my operating system is too old? Are they not compatible? 10.4.11
-
Flash video - mute at start?
I'm using the file ClearOverMute as video controls for my .flv How can I amend ClearOverPlayMute.fla (Action Script 2) so that the initial state is Mute, and the user presses the audio button to hear sound? I'm not a scripter, so clear instructions w