How to extract data from a large table!

Hi all,
I am working on a survey application using JSP, where users login to the application and fill out survey form. Responses of users are stored in a database table as a numeric value ranging from 1-10. Now I want to create some standard reports based on these records. The problem I foresee here is that every survey has almost 15000 user�s records, in this way we have million of records in the table where we keep records. Now while creating reports, if I search the database for these millions of records and then do some calculations/computations on these values, this will turn out a mess. Please suggest what is the best approach to handle the situation.
Thanks

thats something for a Database forum.
But basicly: Use good indexes !

Similar Messages

How to extract data from a pool table?

Hello,
I want to create an generic extractor for table T030, but this is a pool/cluster table and it is not possible to create a view for such a table.
Do somebody know how to solve this problem? Does there exist function modules to read the data from a pool data?
Thanks
Theodor

Function Module would be your best bet..considering thats the best way as far as performance is taken into consideration.
But there is a work around too...
If the other table u use in the join is not a clusture table then u can create a view on that table and u can add append structure to this with fields from the clusture table and use an exit to populate it.
Try it Will Work...used this technique when I used KONV.* This Procedure does not decrease performance considerably.
at least in my case as thre were only a few fields i needed from pool table
Hope this Helps
Anand Raj

JDBC-XI-FILE scenario. How to extract data from more than one table in JDBC

Hi,
I was asked a question like in JDBC-XI-FILE scenario........ How to extract data from more than one tables (i.e from JDBC system) ?? What is the logic to do the same ??
I am not sure whether this is a valid question..........but any help in this regards is highly appreciated.
Regards
Kumar

HI,
Yes it can be possible ,please see the following links
JDBC Receiver with Oracle Stored Procedures
configuring jdbc adapter with multiple tables
RFC -> XI -> JDBC Scenario Updating Multiple Tables
/people/alessandro.berta/blog/2005/10/04/save-time-with-generalized-jdbc-datatypes
JDBC Adapter multiple Selects
https://www.sdn.sap.com/irj/sdn/advancedsearch?query=jdbc%20with%20multiple%20tables&cat=sdn_all
Regards
Chilla..

How to extract data from info cube into an internal table using ABAP code

HI
Can Anyone plz suggest me
How to extract data from info cube into an internal table using ABAP code like BAPI's or function modules.
Thankx in advance
regds
AJAY

HI Dinesh,
Thankq for ur reply
but i ahve already tried to use the function module.
When I try to Use the function module RSDRI_INFOPOV_READ
I get an information message "ERROR GENERATION TEST FRAME".
can U plz tell me what could be the problem
Bye
AJAY

Need suggestion on how to extract data from a table

Many years ago, I wrote many Perl scripts to increase my work productivity.
Now I am starting learning Java, Javascript and PHP for some purposes. Just wrote a simple html2txt java program.
Now I need experts' suggestion for one specific purpose - extract data from a html table. One of the examples is here: http://mops.tse.com.tw/nas/t06sa18/200902/A02_2454_200902.htm
I need to extract datum from the table and make some analysis. (Txt is in Chinese, sorry)
What is the best way to code it? Java, PHP or other? If Java, any suggestion what classes/module and approach to use?
Thanks very much.

xcomme wrote:
Many years ago, I wrote many Perl scripts to increase my work productivity.
Now I am starting learning Java, Javascript and PHP for some purposes. Just wrote a simple html2txt java program.
Now I need experts' suggestion for one specific purpose - extract data from a html table. One of the examples is here: http://mops.tse.com.tw/nas/t06sa18/200902/A02_2454_200902.htm
I need to extract datum from the table and make some analysis. (Txt is in Chinese, sorry)
What is the best way to code it? It this a one shot (one time only task)? Then the best way is whatever way you are most comfortable with and which works.
If on going then I doubt language choice matters but using a html parser rather than attempting to parse it yourself is going to help in any language.
And are you starting with html files or starting with a http server? The two are very different.

How to extract data from custom made Idoc that is not sent

Hi experts,
Could you please advise if there is a way how to extract data from custom made idoc (it collects a lot of data from different SAP tables)? Please note that this idoc is not sent as target system is not fully maintained.
As by now, we would like to verify - what data is extracted now.
Any help, would be appreciated!

Hi,
The fields that are given for each segment have their length given in EDSAPPL table. How you have to map is explained in below example.
Suppose for segment1, EDSAPPL has 3 fields so below are entries
SEGMENT          FIELDNAME           LENGTH
SEGMENT1         FIELD1                   4
SEGMENT1         FIELD2                   2
SEGMENT1         FIELD3                   2
Data in EDID4 would be as follows
IDOC           SEGMENT                          APPLICATION DATA
12345         SEGMENT1                        XYZ R Y
When you are extracting data from these tables into your internal table, mapping has to be as follows:
FIELD1 = APPLICATIONDATA+0(4)        to read first 4 characters of this field, because the first 4 characters in this field would belong to FIELD1
Similarly,
FIELD2 = APPLICATIONDATA+4(2).
FIELD3 = APPLICATIONDATA+6(2).
FIELD1 would have XYZ, FIELD2 = R, FIELD3 = Y
This would remain true in all cases. So all you need to do is identify which fields you want to extract, and simply code as above to extract the data from this table.
Hope this was helpful in explaining how to derive the data.

How to move data from a staging table to three entity tables #2

Environment: SQL Server 2008 R2
I have a few questions:
How would I prevent duplicate records, when/ IF SSIS is executed many times?
How would I know that all huge volume of data being loaded in the entity tables?
In reference to "how to move data from a staging table to three entity tables ", since I am loading large volume of data, while using lookup transformation:
which of the merge components is best suited.
How to configure merge component correctly. (screen shot is preferred)
Please refer to the following link
http://social.msdn.microsoft.com/Forums/en-US/5f2128c8-3ddd-4455-9076-05fa1902a62a/how-to-move-data-from-a-staging-table-to-three-entity-tables?forum=sqlintegrationservices

You can use RowCount transformation in the path where you want to capture record details. Then inside rowcount transformation pass a integer variable to get count value inside
the event handler can be configured as below
Inside Execute SQL task add INSERT statement to add rowcount to your audit table
Can you also show me how to Check against destination table using key columns inside a lookup task and insert only non
matched records (No Match output)
This is explained clearly in below link which Arthur posted
http://www.sqlis.com/sqlis/post/Get-all-from-Table-A-that-isnt-in-Table-B.aspx
For large data I would prefer doing this in T-SQL. So what you could do is dump data to staging table and then apply
T-SQL MERGE between tables (or even a combination of INSERT/UPDATE statements)
Please Mark This As Answer if it helps to solve the issue Visakh ---------------------------- http://visakhm.blogspot.com/ https://www.facebook.com/VmBlogs

CALL function to extract data from a structure table

Hi Gurus,
Anyone knows how to write a call function to extract data from a structure table?
Your help is very much appreciated.
Thanks alot.

Hi,
structure doesnot hold any data. instead of it you can check the stucture in which table it is used, find the table name,use select query to extract the data u needed.
you can use where used list option to find the structure in which table it is used.
regards
siva

How to extract data From Hyperion Essbase to Flat Files

Hi ,
Can anyone tell me how to extract data from Hyperion essbase to Flat file(e.g. .csv , .txt ,...) or Oracle Table using ODI.
Regards,
Avneet

Hi sutirtha,
I have make one column as key column now i am getting this error,
org.apache.bsf.BSFException: exception from Jython:
Traceback (innermost last):
File "<string>", line 1, in ?
com.hyperion.odi.essbase.ODIEssbaseException: The number of columns returned by script [10] is less than the source data columns exposed [12]
at com.hyperion.odi.essbase.ODIEssbaseDataReader.getAppData(Unknown Source)
at com.hyperion.odi.essbase.AbstractEssbaseReader.extract(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java)
at org.python.core.PyMethod.__call__(PyMethod.java)
at org.python.core.PyObject.__call__(PyObject.java)
at org.python.core.PyInstance.invoke(PyInstance.java)
at org.python.pycode._pyx7.f$0(<string>:1)
at org.python.pycode._pyx7.call_function(<string>)
at org.python.core.PyTableCode.call(PyTableCode.java)
at org.python.core.PyCode.call(PyCode.java)
at org.python.core.Py.runCode(Py.java)
at org.python.core.Py.exec(Py.java)
at org.python.util.PythonInterpreter.exec(PythonInterpreter.java)
at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:144)
at com.sunopsis.dwg.codeinterpretor.k.a(k.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.scripting(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTaskTrt(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSqlC.treatTaskTrt(SnpSessTaskSqlC.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java)
at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java)
at com.sunopsis.dwg.cmd.DwgCommandSession.treatCommand(DwgCommandSession.java)
at com.sunopsis.dwg.cmd.DwgCommandBase.execute(DwgCommandBase.java)
at com.sunopsis.dwg.cmd.e.k(e.java)
at com.sunopsis.dwg.cmd.g.A(g.java)
at com.sunopsis.dwg.cmd.e.run(e.java)
at java.lang.Thread.run(Unknown Source)
Caused by: com.hyperion.odi.essbase.ODIEssbaseException: The number of columns returned by script [10] is less than the source data columns exposed [12]
at com.hyperion.odi.essbase.wrapper.EssbaseReportDataIterator.validateColumns(Unknown Source)
at com.hyperion.odi.essbase.wrapper.EssbaseReportDataIterator.init(Unknown Source)
... 33 more
com.hyperion.odi.essbase.ODIEssbaseException: com.hyperion.odi.essbase.ODIEssbaseException: The number of columns returned by script [10] is less than the source data columns exposed [12]
at org.apache.bsf.engines.jython.JythonEngine.exec(JythonEngine.java:146)
at com.sunopsis.dwg.codeinterpretor.k.a(k.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.scripting(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.execScriptingOrders(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTaskTrt(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSqlC.treatTaskTrt(SnpSessTaskSqlC.java)
at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java)
at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java)
at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java)
at com.sunopsis.dwg.cmd.DwgCommandSession.treatCommand(DwgCommandSession.java)
at com.sunopsis.dwg.cmd.DwgCommandBase.execute(DwgCommandBase.java)
at com.sunopsis.dwg.cmd.e.k(e.java)
at com.sunopsis.dwg.cmd.g.A(g.java)
at com.sunopsis.dwg.cmd.e.run(e.java)
at java.lang.Thread.run(Unknown Source)

Retrieve data from a large table from ORACLE 10g

I am working with a Microsoft Visual Studio Project that requires to retrieve data from a large table from Oracle 10g database and export the data into the hard drive.
The problem here is that I am not able to connect to the database directly because of license issue but I can use a third party API to retrieve data from the database. This API has sufficient previllege/license permission on to the database to perform retrieval of data. So, I am not able to use DTS/SSIS or other tool to import data from the database directly connecting to it.
Here my approach is...first retrieve the data using the API into a .net DataTable and then dump the records from it into the hard drive in a specific format (might be in Excel file/ another SQL server database).
When I try to retrieve the data from a large table having over 13 lacs records (3-4 GB) in a data table using the visual studio project, I get an Out of memory exception.
But is there any better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?
Any help on this problem will be highly appriciated.
Thanks in advance...
-Jahedur Rahman
Edited by: Jahedur on May 16, 2010 11:42 PM

Girish...Thanks for your reply...But I am sorry for the confusions. Let me explain that...
1."export the data into another media into the hard drive."
What does it mean by this line i.e. another media into hard drive???
ANS: Sorry...I just want to write the data in a file or in a table in SQL server database.
2."I am not able to connect to the database directly because of license issue"
huh?? I never heard this question that a user is not able to connect the db because of license. What error / message you are getting?
ANS: My company uses a 3rd party application that uses ORACLE 10g. And my compnay is licensed to use the 3rd party application (APP+Database is a package) and did not purchased ORACLE license to use directly. So I will not connect to the database directly.
3.I am not sure which API is you are talking about, but i am running an application of the visual studio data grid or similar kind of controls; in which i can select (select query) as many rows as i needed; no issue.
ANS: This API is provided by the 3rd party application vendor. I can pass a query to it and it returns a datatable.
4."better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?"
ANS: As I get a system error (out of memory) when I select all rows in a datatable at a time, I wanted to retrieve the data in multiple phases.
E.g: 1 to 20,000 records in 1st phase
20,001 to 40,000 records in 2nd phase
40,001 to ...... records in 3nd phase
and so on...
Please let me know if this does not clarify your confusions... :)
Thanks...
-Jahedur Rahman
Edited by: user13114507 on May 12, 2010 11:28 PM

How to extract data from SAP in FDM 11121

I came across some documents from version 11113, says there is a SAP adapter, however ,when I check 11121 there's no such adapter, does anyone know how to extract data from SAP in FDM 11121?

I download a package from Bristlecone, but I dont see any xml files in it, just a bunch of dll and I didn't find any instructions on how to set up/configure in FDM, it just explained how to register in SAP app server and client.
Hyperion 11113 has readme on sap adapter(Hyperion Readme Template), but I cannot find the same readme file in 11121, all I can find is ERP Integration Adapter document. any idears where I can find at least a readme document?

How to extract data from web URL

I was doing one project which need to extract data from web pages and then analyze these data. the question is how to extract data from there, using html parser? need help, thanks a lot

I was doing one project which need to extract data
from web pages and then analyze these data. the
question is how to extract data from there, using
html parser? need help, thanks a lotTry this:
http://java.sun.com/docs/books/tutorial/networking/urls/readingURL.html
Or, like you said yourself, use an HTML parser:
http://java-source.net/open-source/html-parsers

How to extract data from Chart History?

Dear all, I have read a lot of posts, but still don't understand how to extract data from Chart history.
Suppose you acquired 1024 points of data every time, then use "build array" to build an array, then send to intensity chart to show, then save the history into a .txt file, but how to extact data from the file?
How Labview store the data in the 2D history buffer?
Anybody has any examples?

The simplest would be to save the 2D array as a spreadsheet file, the read it back the same way.
Maybe the attached simple example can give you some ideas (LabVIEW 7.1). Just run it. At any time, press "write to file". At any later time, you can read the save data into the second history chart.
If you are worried about performance, it might be better to use binary files, but it will be a little more complicated. See how far you get.
LabVIEW Champion . Do more with less code and in less time .
Attachments:
IntensityChartHistorySave.vi ‏79 KB

How to extract data from planning book

nu t apo. need help regarding how to extract data from planning book givn the planning book name , view name & some key figures.
Total Demand (Key Figure DMDTO):
o     Forecast
o     Sales Order
o     Distribution Demand (Planned)
o     Distribution Demand (Confirmed)
o     Distribution Demand (TLB-Confirmed)
o     Dependent Demand
Total Receipts (Key Figure RECTO):
o     Distribution Receipt (Planned)
o     Distribution Receipt (Confirmed)
o     Distribution Receipt (TLB-Confirmed)
o     In-Transit
o     Production (Planned)
o     Production (Confirmed)
o     Manufacture of Co-Products
Stock on Hand (Key Figure STOCK):
o     Stock on Hand (Excluding Blocked stock)
o     Stock on Hand (Including Blocked stock)
In-Transit Inventory (Cross-company stock transfers)
Work-in-progress (Planned orders / Process orders with start date in one period and finish date in another period)
Production (Planned), Production (Confirmed) and Distribution Receipt elements need to be converted based on Goods Receipt date for projected inventory calculation.

Hello Debadrita,
Function Module BAPI_PBSRVAPS_GETDETAIL2 or BAPI_PBSRVAPS_GETDETAIL can help you.
For BAPI BAPI_PBSRVAPS_GETDETAIL, the parameters are:
1) PLANNINGBOOK - the name of your planning book
2) DATA_VIEW - name of your data view
3) KEY_FIGURE_SELECTION - list of key figures you want to read
4) SELECTION - selection parameters which describe the attributes of the data you want to read (e.g. the category or brand). This is basically a list of characteristics and characteristic values.
BAPI_PBSRVAPS_GETDETAIL2 is very similar to BAPI_PBSRVAPS_GETDETAI but is only available from SCM 4.1 onwards.
For the complete list of parameters, you can go to transaction SE37, enter the function module names above and check out the documentation.
Please post again if you have questions.
Hope this helps.

How to extract data from ODS to non-SAP system

Hi,
Can anybody tell me, step by step, how to extract data from ODS to a non-SAP system?
Is it possible to do it without programming effort? And is there volume limits for this kind of extraction?
The non-SAP system is an unix system.
Thanks in advance
Ella

Ella,
You can look at it from the concept of a BADI / Infospoke
Extract the data from the ODS to a Flat file / RDBMS using an infospoke. I am not sure as to how the infospoke loads data into the RDBMS ( did it very long ago ) but then you can push it into an RDBMS and I am sure it will be system neutral.
Hope this helps...
Arun
Assign points if it helps

How to extract data from a large table!

Similar Messages

Maybe you are looking for