Fetching data from a large table

hi
I am trying to fetch data from a table with 100 million rows. There are 5 conditions(AND) in the "WHERE" clause. Out of five columns there is composite non unique index on 2 columns. rest of the three columns are haing data with low cardinality.i just wana know what type of index is suggested for my query so that i can fetch the data immediately i.e. should i go for composite index on all the 5 columns. secondly there is continuous insertion also. if i create index on all the 5 columns how it effect my insertion.i am using 8i
Thanx
Tarun

This forum is for posting feedback about the OTN site.
The best place to get an answer to your question is a Database forum, perhaps the PL/SQL forum.

Similar Messages

Retrieve data from a large table from ORACLE 10g

I am working with a Microsoft Visual Studio Project that requires to retrieve data from a large table from Oracle 10g database and export the data into the hard drive.
The problem here is that I am not able to connect to the database directly because of license issue but I can use a third party API to retrieve data from the database. This API has sufficient previllege/license permission on to the database to perform retrieval of data. So, I am not able to use DTS/SSIS or other tool to import data from the database directly connecting to it.
Here my approach is...first retrieve the data using the API into a .net DataTable and then dump the records from it into the hard drive in a specific format (might be in Excel file/ another SQL server database).
When I try to retrieve the data from a large table having over 13 lacs records (3-4 GB) in a data table using the visual studio project, I get an Out of memory exception.
But is there any better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?
Any help on this problem will be highly appriciated.
Thanks in advance...
-Jahedur Rahman
Edited by: Jahedur on May 16, 2010 11:42 PM

Girish...Thanks for your reply...But I am sorry for the confusions. Let me explain that...
1."export the data into another media into the hard drive."
What does it mean by this line i.e. another media into hard drive???
ANS: Sorry...I just want to write the data in a file or in a table in SQL server database.
2."I am not able to connect to the database directly because of license issue"
huh?? I never heard this question that a user is not able to connect the db because of license. What error / message you are getting?
ANS: My company uses a 3rd party application that uses ORACLE 10g. And my compnay is licensed to use the 3rd party application (APP+Database is a package) and did not purchased ORACLE license to use directly. So I will not connect to the database directly.
3.I am not sure which API is you are talking about, but i am running an application of the visual studio data grid or similar kind of controls; in which i can select (select query) as many rows as i needed; no issue.
ANS: This API is provided by the 3rd party application vendor. I can pass a query to it and it returns a datatable.
4."better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?"
ANS: As I get a system error (out of memory) when I select all rows in a datatable at a time, I wanted to retrieve the data in multiple phases.
E.g: 1 to 20,000 records in 1st phase
20,001 to 40,000 records in 2nd phase
40,001 to ...... records in 3nd phase
and so on...
Please let me know if this does not clarify your confusions... :)
Thanks...
-Jahedur Rahman
Edited by: user13114507 on May 12, 2010 11:28 PM

How to fetch data from single database table using 2 internal tables.

Hi friends,
i am a new user of ABAP and also SDN.
i need a help.
i want to fetch data from one database table based on primary keys of 2 internal tables. how to put in where clause.
Thanks in advance.

hii
refer to following code ..i hope it will help you
SELECT matnr                         " Material Number
    FROM mara
    INTO TABLE i_mara
   WHERE matnr IN s_matnr.
IF i_mara[] IS NOT INITIAL.
    SELECT matnr                       " Material Number
           werks                       " Plants
           prctr                       " Profit Center
      FROM marc
      INTO TABLE i_marc
       FOR ALL ENTRIES IN i_mara
     WHERE matnr = i_mara-matnr
       AND werks IN s_werks.
ENDIF.                               " IF i_mara[] IS NOT INITIAL
i_output = i_marc.
IF i_marc[] IS NOT INITIAL.
    SELECT matnr                       " Material Number
           werks                       " Plants
           lgort                       " Storage Location
      FROM mard
      INTO TABLE i_mard
       FOR ALL ENTRIES IN i_marc
     WHERE matnr EQ i_marc-matnr
       AND werks EQ i_marc-werks
       AND lgort IN s_lgort.
ENDIF.                               " IF i_mara[] IS NOT INITIAL
regards
twinkal

Fetch data from different database tables

Hi...
How can i fetch data from different database tables and put it into a internal table and then display it??? Can provide simple short codes as i'm new to ABAP. Thanks.

Hi,
Check this sample code..
TYPE-POOLS: slis.
DATA: BEGIN OF itab OCCURS 0,
        vbeln TYPE vbeln,
        expand,
      END OF itab.
DATA: BEGIN OF itab1 OCCURS 0,
        vbeln TYPE vbeln,
        posnr TYPE posnr,
        matnr TYPE matnr,
        netpr TYPE netpr,
      END OF itab1.
DATA: t_fieldcatalog TYPE slis_t_fieldcat_alv.
DATA: s_fieldcatalog TYPE slis_fieldcat_alv.
s_fieldcatalog-col_pos = '1'.
s_fieldcatalog-fieldname = 'VBELN'.
s_fieldcatalog-tabname   = 'ITAB'.
s_fieldcatalog-rollname = 'VBELN'.
s_fieldcatalog-outputlen = '12'.
APPEND s_fieldcatalog TO t_fieldcatalog.
CLEAR: s_fieldcatalog.
s_fieldcatalog-col_pos = '1'.
s_fieldcatalog-fieldname = 'VBELN'.
s_fieldcatalog-tabname   = 'ITAB1'.
s_fieldcatalog-rollname = 'VBELN'.
s_fieldcatalog-outputlen = '12'.
APPEND s_fieldcatalog TO t_fieldcatalog.
CLEAR: s_fieldcatalog.
s_fieldcatalog-col_pos = '2'.
s_fieldcatalog-fieldname = 'POSNR'.
s_fieldcatalog-tabname   = 'ITAB1'.
s_fieldcatalog-rollname = 'POSNR'.
APPEND s_fieldcatalog TO t_fieldcatalog.
CLEAR: s_fieldcatalog.
s_fieldcatalog-col_pos = '3'.
s_fieldcatalog-fieldname = 'MATNR'.
s_fieldcatalog-tabname   = 'ITAB1'.
s_fieldcatalog-rollname = 'MATNR'.
APPEND s_fieldcatalog TO t_fieldcatalog.
CLEAR: s_fieldcatalog.
s_fieldcatalog-col_pos = '4'.
s_fieldcatalog-fieldname = 'NETPR'.
s_fieldcatalog-tabname   = 'ITAB1'.
s_fieldcatalog-rollname = 'NETPR'.
s_fieldcatalog-do_sum    = 'X'.
APPEND s_fieldcatalog TO t_fieldcatalog.
CLEAR: s_fieldcatalog.
DATA: s_layout TYPE slis_layout_alv.
s_layout-subtotals_text            = 'SUBTOTAL TEXT'.
s_layout-key_hotspot = 'X'.
s_layout-expand_fieldname = 'EXPAND'.
SELECT vbeln UP TO 100 ROWS
       FROM
       vbak
       INTO TABLE itab.
IF NOT itab[] IS INITIAL.
SELECT vbeln posnr matnr netpr
         FROM vbap
         INTO TABLE itab1
         FOR ALL ENTRIES IN itab
         WHERE vbeln = itab-vbeln.
ENDIF.
DATA: v_repid TYPE syrepid.
v_repid = sy-repid.
DATA: s_keyinfo TYPE slis_keyinfo_alv.
s_keyinfo-header01 = 'VBELN'.
s_keyinfo-item01   = 'VBELN'.
CALL FUNCTION 'REUSE_ALV_HIERSEQ_LIST_DISPLAY'
     EXPORTING
          i_callback_program = v_repid
          is_layout          = s_layout
          it_fieldcat        = t_fieldcatalog
          i_tabname_header   = 'ITAB'
          i_tabname_item     = 'ITAB1'
          is_keyinfo         = s_keyinfo
     TABLES
          t_outtab_header    = itab
          t_outtab_item      = itab1
     EXCEPTIONS
          program_error      = 1
          OTHERS             = 2.
IF sy-subrc <> 0.
* MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
*         WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
ENDIF.
Thanks
Naren

Authorization control when fetching data from PA OM tables

Hi all,
I want to know how to control DB level authorization using HR_READ_INFOTYPE and RP_READ_INFOTYPE. Taht is, i want to check whetehr the user who is executing a report is having read authorization for that infotype. If not, i should not fetch the data and the report should return error message?
Instead of authorization (std, custom) object, i want to control when fetching data from PA and OM tables in HR. How to achieve the same?

As EJP said, your query might have problem with the using clause. Post your query.
It gives an exception when I try to run the query, and I'm trying to fetch all columns not only 'taskid'
java.sql.SQLException: ORA-25154: column part of USING clause cannot have qualifier
Yes, with the exact sql query he posted. It gives that exception.I guess you are trying to fetch all the columns by using the above query which i posted. No you should not fetch all the columns. Because the rs will only the three columns.
Also, use 'as' with that query like,
select t1.taskid as t1_taskid, t2.taskid as t2_taskid ,t3.taskid as t3_taskid from...Edited by: Ram on Aug 23, 2011 6:20 PM

Fetch data from a R3 Table and send to XI

Hello XI Experts,
I am involved in a scenario where i need to read some data from a Transparent table in R3 send to xi and post a CSV File.
I have worked on look ups at the XI abap stack but i want to know which is the best solution to do this.
What's the best way to read a table from R/3 and send to xi server. Is there any other ways besides Look ups?
Regards,
Anirban.

Hi,
>>>1. When is look ups advised .
a lookup is when somewhere in your
interface flow you have to lookup some data
for example you send data from one system to another but you have to change material numbers before
sending the data to the target system
you know that material numbers are inside some other server so you have to call this sever (very often in a sync way) to lookup the material numbers
so this would be difficult with IDOC for instance
>>> 2. Which is a better solution, if i have very few data to fetch with WAS >= 6.20 . Abap Proxy Or a 'Z' Idoc.
you'll be able to do it quicker with ABAP proxy
if you use a scheduled report that will send the data from r3 then use ABAP proxies
Regards,
michal

To fetch Data from multiple database tables!

How to fetch Data from fields of multiple database tables!
Give me one example!

use <b>join....</b>
c the SAPHELP docs...
FROM tabref1 [INNER] JOIN tabref2 ON cond
Effect
The data is to be selected from transparent database tables and/or views determined by tabref1 and tabref2. tabref1 and tabref2 each have the same form as in variant 1 or are themselves Join expressions. The keyword INNER does not have to be specified. The database tables or views determined by tabref1 and tabref2 must be recognized by the ABAP Dictionary.
In a relational data structure, it is quite normal for data that belongs together to be split up across several tables to help the process of standardization (see relational databases). To regroup this information into a database query, you can link tables using the join command. This formulates conditions for the columns in the tables involved. The inner join contains all combinations of lines from the database table determined by tabref1 with lines from the table determined by tabref2, whose values together meet the logical condition (join condition) specified using ON>cond.
Inner join between table 1 and table 2, where column D in both tables in the join condition is set the same:
Table 1                      Table 2
A
B
C
D
D
E
F
G
H
a1
b1
c1
1
1
e1
f1
g1
h1
a2
b2
c2
1
3
e2
f2
g2
h2
a3
b3
c3
2
4
e3
f3
g3
h3
a4
b4
c4
3
|--|||--|
    Inner Join
    |--||||||||--|
    | A | B | C | D | D | E | F | G | H |
    |--||||||||--|
    | a1 | b1 | c1 | 1 | 1 | e1 | f1 | g1 | h1 |
    | a2 | b2 | c2 | 1 | 1 | e1 | f1 | g1 | h1 |
    | a4 | b4 | c4 | 3 | 3 | e2 | f2 | g2 | h2 |
    |--||||||||--|
Example
Output a list of all flights from Frankfurt to New York between September 10th and 20th, 2001 that are not sold out:
DATA: DATE   LIKE SFLIGHT-FLDATE,
      CARRID LIKE SFLIGHT-CARRID,
      CONNID LIKE SFLIGHT-CONNID.
SELECT FCARRID FCONNID F~FLDATE
    INTO (CARRID, CONNID, DATE)
    FROM SFLIGHT AS F INNER JOIN SPFLI AS P
           ON FCARRID = PCARRID AND
              FCONNID = PCONNID
    WHERE P~CITYFROM = 'FRANKFURT'
      AND P~CITYTO   = 'NEW YORK'
      AND F~FLDATE BETWEEN '20010910' AND '20010920'
      AND FSEATSOCC < FSEATSMAX.
WRITE: / DATE, CARRID, CONNID.
ENDSELECT.
If there are columns with the same name in both tables, you must distinguish between them by prefixing the field descriptor with the table name or a table alias.

Open Hub fetching data from Change log table

Hi Gurus,
I have a scenario where in I have an Open Hub destination whose source is a DSO, and we are sending data into SAP Data services. We found that the data being sent is from the change log table. Is this how an Open Hub with DSO as a source works ? Or is there any possible way we can send data from the Active table of the DSO, when the DTP is run.
Kindly help. Thanks in advance !
Regards
Snehith

Hi,
For Extraction from the DataStore Object, we have the following options for Delta init Extraction:
Active Table (with Archive)
The data is read from the active table and from the archive or from a near-line storage if one exists. You can choose this option even if there is no active data archiving process yet for the DataStore object.
Active Table (Without Archive)
The data is only read from the active table. If there is data in the archive or in a near-line storage at the time of extraction, this data is not extracted.
Change Log
The data is read from the change log of the DataStore object
We can select any of these options for data trasnfer.
Regards,
Geetanjali

Java.lang.OutOfMemory error while retrieving data from a large table

Hi,
i am trying to fetch data using "executeQuery()" into a ResultSet from the database. But since the data in that table is large. i am recieving "java.lang.OutOfMemory" Error. So, to resolve that, i have used "setMaxRows()" for my statement object. This resolved the error but i don't recieve the entire data. If i call "executeQuery()" again, i recieve the same data. I don't even know a filtering criterion where by i can filter the data for each "executeQuery()"..
How can i resolve this problem
Thanx in advance
--Chaitanya

Either use some criteria you develop related to one of the keys on the table or use some sort of record limiting method.
Note the method of limiting will vary related to the database you are using. You will have to look at the documentation.
For example I am told this will work in MySQL to get 200 records starting at record 100.
SELECT * FROM myTable ORDER BY whatever ASC LIMIT 100,200
Because you are running out of memroy I assume the table is large,
I am not sure what the impact of the above will have on performance because if in the above if the order by is not based on an index at the server level all the records will be selected and sorted before the records are limited.
I would make sure you have an appropriate index.
If you use the advanced search over the user forums using "resultset paging" and possibility the database you are using you should be able to get some ideas.
I hope this makes sense to you.
rykk

How to fetch data from an internal table by comparing the a value in loop

i have an internale table with two fields like parvw and vtext.
acc to valuue in parvw in loop we sholud fetch the corresponding value of vtext.
we should not use loop in loop .
ex code
select parvw vtext from tpart into table i_vtext where spras = sy-langu.
    LOOP AT I_OUTPUT.
      CASE I_OUTPUT-tabname.
        WHEN 'VBPA'.
          IF I_OUTPUT-fname = 'KUNNR' OR
             I_OUTPUT-fname IS INITIAL.
            MOVE I_OUTPUT-tabkey TO vbpa.
           READ TABLE i_vtext WITH table KEY parvw = vbpa-*parvw .
            IF sy-subrc = 0.
              REPLACE '&' WITH i_vtext-vtext INTO I_OUTPUT-indtext.
            ENDIF.
          ENDIF.
        WHEN 'VBAP'.
          IF I_OUTPUT-fname IS INITIAL.
            REPLACE '&' WITH 'item' INTO I_OUTPUT-indtext.
          ENDIF.
      ENDCASE.
      IF I_OUTPUT-indtext(1) EQ '&'.
        REPLACE '&' WITH I_OUTPUT-ftext(40) INTO I_OUTPUT-indtext.
      ENDIF.

Loop the main ITAB
and then use READ TABLE to fetch a row from the 2nd ITAB.
sort the main itab abd use binary search option in READ to improve the performance
Narendra

2LIS_04_P_ARBPL - how this extractor fetching data from which all tables??

Hi
We have one DataSource 2LIS_04_P_ARBPL and we want to know all the tables of origin.
Actually the problem is: this datasource is extracting a blank workcenter
Suppose we have one Process Order 100000 for which this DS is giving us 4 records: out of this 4 records, 1 record is having blank workcenter and that's why we want to know how this DS 2LIS_04_P_ARBPL is extracting data from the tables.
I tried to run RSA3 in debug mode but it couldnt help me out much
Also I ran trace ST05 and tried to find out tables involved,,but this method also failed
I searched in help.sap.com but again I ended up with disappointment:
http://help.sap.com/saphelp_nw04/helpdata/en/6c/61117e6c3911d3b6df0000e8359890/content.htm
As its logistics data source so i used LBWE but at last this mehtod also not worked out.
Request you all to help me out on this issue.
Thanks
Alok

well if it was such a simple things then our extractor will give only 1 record as AFKO/AFPO table returns only 1 record for the given Process order but here we are getting 4 records.
u can try in yr system as well. run RSA3 for same DS and pick any Process order . u will definately find more than 1 entry but AFKO/AFPO will give u one entry , especially the header table AFKO

Fetching data from one internal table and populating into other table

hi
i am working on a report program.
here,i have my data in a internal table(having only one field) i.e. only one column in the table.
now i want to populate this data into other internal internal table that has 13fields i.e. 13 columns.
so could anyone let me know how can i work on this.
i am looping the internal table with one field into a work area from there i am not able to proceed ahead.
thanks in advance

hi
i would like to clear one point here i.e. my internal table into which i want to populate data is a dynamic internal table....so i cant give the fieldnames here..
here i am pasting the code..so that u may get some idea...
"generating dynamic internal table :
ref_descr ?= cl_abap_typedescr=>describe_by_data( it_itab ).
it_details[] = ref_descr->components[].
loop at it_details into wa_comp.
cnt = cnt + 1.
wa_fcat-col_pos = cnt.
wa_fcat-fieldname = wa_comp-name.
append wa_fcat to t_fcat.
endloop.
CALL METHOD CL_ALV_TABLE_CREATE=>CREATE_DYNAMIC_TABLE
EXPORTING
    I_STYLE_TABLE             =
    IT_FIELDCATALOG           = t_fcat
    I_LENGTH_IN_BYTE          =
IMPORTING
   EP_TABLE                  = lo_table
    E_STYLE_FNAME             =
EXCEPTIONS
    GENERATE_SUBPOOL_DIR_FULL = 1
    others                    = 2
so lo_table is my dynamic internal table.
ASSIGN lo_table->* TO <f_tab>.
using the GUI_UPLOAD i have uploaded the data file(separated by semicolon) from presentation server and its in the table itab.
with the following logic i am capturing the data in between the semicolon's.
LOOP AT itab INTO wa_itab.
find all occurrences of c_semicolon in wsg_string results result_tab.
loop at result_tab into wa_reslt.
if sy-tabix = 1.
str1 = wsg_string+0(wa_reslt-offset).
wa_prev_offset = wa_reslt-offset.
wa_prev_offset = wa_prev_offset + 1.
clear str1.
else.
   count = wa_reslt-offset - wa_prev_offset.
   count = count.
str2 = wsg_string+wa_prev_offset(count).
clear str2.
wa_prev_offset = wa_reslt-offset.
wa_prev_offset = wa_prev_offset + 1.
enddo.
endif.
endloop.
data is there in the fields str1 and str2.now i want to populate this data into dynamic internal table row-wise.
could anyone suggest me how can i do this

How to fetch data from different standard table to own customize table.

Hi Experts,
I want to develope an INVENTORY TURN OVER REPORT in SAP R/3.As I dont know much about Inventory.I want some one here to guide me regarding this with example so i can develope Inventory Report in R/3.
Also guide me which filed from which table i shud use.What is Inventory Turnover in details and how it is calculated.
Plz reply guys.
Thanks.
Edited by: abhishek.chaturvedi on Aug 9, 2010 8:42 AM

Hi,
Even iam not sure about it but after looking at your query even i have done some research and got some information for you.
nventory Turnover is a prominent KPI in a lot of businesses. This post will give you some understanding on the concept. In another post I will give details on how the SAP report MC44 can be used to measure inventory turns and what data is used to calculate this ratio.
Formula
Inventory Turnover = Cost of Goods Sold (COGS) / Average Inventory at value
If you are not familiar with the term Cost of Goods Sold, this is the cost of your revenues.
Average Inventory is measured in value and not in volume.
If you divide both elements you know how often you sold youu2019re average inventory. u201CSo what?u201D you ask. I will explain next.
Concept and use
I will present you with two business scenariou2019s to point out the significance of the Inventory Turnover ratio.
Scenario 1
You sell one product. You invest 10.000 EUR as starting stock. You run your business for one year and at then end of the year you are completely sold out. Your revenue accumulates to 12.500 EUR. For simplicity sake your gross profit is 2.500 EUR (12.500 u2013 10.000).
Revenue = 12.500
COGS = 10.000
Average Inventory = 5.000 (10.000 starting stock + 0 end stock / 2 = 5.000 average stock)
Inventory Turnover = 2 (10.000 / 5.000)
ROI = 50% > 2.500 (R-C) earned out of 5.000 (A)
Scenario 2
Again you sell one product. But this time you invest 5.000 EUR in starting stock. After 6 months you sold your stock and you replenish your stock again for 5.000 EUR. After another 6 months youu2019re sold out. Your revenue is again 12.500 EUR. Your gross profit is 2.500 EUR (12.500 u2013 5000 u2013 5000).
Revenue = 12.500
COGS = 10.000
Average Inventory = 2.500 (5.000 starting stock + 0 end stock / 2 = 2.500 average stock)
Inventory Turnover = 4 (10.000 / 2.500)
ROI = 100% > 2.500 (R-C) earned out of 2.500 (A)
cheers,
bhavana

How to fetch data from nested internal table

Hi,
Im using FM CRM_PRODUCT_GETDETAIL_API which is returning me work area (ES_PRODUCT_DATA) of type
COMT_PRODUCT_MAINTAIN_API. This work area contains a table SHORT_TEXTS of type COMT_PR_SHTEXT_MAINTAIN_TAB
whcih in turn contain a line type DATA of type COMT_PRSHTEXT_X. I need to fetch fields lying inside DATA. Can anybody please
let me know how to achieve this. I would like to do this by not using nested loop structure.
Rgds
Sudhanshu Sharma

Hi,
Use field ES_PRODUCT_DATA-short_texts like any other internal table (without header line).
E.g.:
field-symbols: <text> type COMT_PRSHTEXT_X.
loop at ES_PRODUCT_DATA-short_texts assigning <text>.
write: <text>-short_text
endloop.
Regards, Gerd Rother

How to fetch data from multiple fact tables from a normalized schema?

Hello everybody,
I am working on a normalized schema to build my repository. I have categorized the tables into dimensions and facts. I have 3 fact tables in my schema and they have a 1:m:1 relationship i.e. if I have table A, B, and C; A has 1:m relationship with B and C also has 1:m relationship with B. How can I use measures from these tables to create a star schema. If A = Sales, B = Transaction, C = Payment_Amount
Sales(Sales ID, Amount, Tax pct, ...)
Transaction(Transaction ID, Sales ID, Payment ID,Transaction Amt, ...)
Payment_Amount(Payment ID, Check ID, Payment Made, ...)
Please give me some direction to pursue.
Thank you!
D

Hi dcole,
Go through this link for snowflake schema http://gerardnico.com/wiki/datamodeling/snowflake_
http://www.rittmanmead.com/2007/06/19/obiee-data-modeling-tips-1-integrating-1-1-and-1-many-source-tables/
I suppose it should work with snowflake schema...Im interested in this topic please let me know what is your approach every now and then.
By,
KK

Fetching data from a large table

Similar Messages

Maybe you are looking for