Performance question - Caching data of a big table

Hi All,
I have a general question about caching, I am using an Oracle 11g R2 database.
I have a big table about 50 millions of rows that is accessed very often by my application. Some query runs slow and some are ok. But (obviously) when the data of this table are already in the cache (so basically when a user requests the same thing twice or many times) it runs very quickly.
Does somebody has any recommendations about caching the data / table of this size ?
Many thanks.

Chiwatel wrote:
With better formatting (I hope), sorry I am not used to the new forum !
Plan hash value: 2501344126
| Id | Operation                            | Name          | Starts | E-Rows |E-Bytes| Cost (%CPU)| Pstart| Pstop | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
| 0 | SELECT STATEMENT        |                    |      1 |        |      | 7232 (100)|      |      | 68539 |00:14:20.06 |    212K| 87545 |      |      |          |
| 1 | SORT ORDER BY                      |                |      1 | 7107 | 624K| 7232 (1)|      |      | 68539 |00:14:20.06 |    212K| 87545 | 3242K| 792K| 2881K (0)|
2 | NESTED LOOPS                      |                |      1 |        |      |            |      |      | 68539 |00:14:19.26 |    212K| 87545 |      |      |          |
| 3 |    NESTED LOOPS                      |                |      1 | 7107 | 624K| 7230 (1)|      |      | 70492 |00:07:09.08 |    141K| 43779 |      |      |          |
* 4 |    INDEX RANGE SCAN                | CM_MAINT_PK_ID |      1 | 7107 | 284K|    59 (0)|      |      | 70492 |00:00:04.90 |    496 |    453 |      |      |          |
| 5 |    PARTITION RANGE ITERATOR        |                | 70492 |      1 |      |    1 (0)| KEY | KEY | 70492 |00:07:03.32 |    141K| 43326 |      |      |          |
|* 6 |      INDEX UNIQUE SCAN              | D1T400P0      | 70492 |      1 |      |    1 (0)| KEY | KEY | 70492 |00:07:01.71 |    141K| 43326 |      |      |          |
|* 7 |    TABLE ACCESS BY GLOBAL INDEX ROWID| D1_DVC_EVT    | 70492 |      1 |    49 |    2 (0)| ROWID | ROWID | 68539 |00:07:09.17 | 70656 | 43766 |      |      |          |
Predicate Information (identified by operation id):
4 - access("ERO"."MAINT_OBJ_CD"='D1-DEVICE' AND "ERO"."PK_VALUE1"='461089508922')
6 - access("ERO"."DVC_EVT_ID"="E"."DVC_EVT_ID")
7 - filter(("E"."DVC_EVT_TYPE_CD"='END-GSMLOWLEVEL-EXCP-SEV-1' OR "E"."DVC_EVT_TYPE_CD"='STR-GSMLOWLEVEL-EXCP-SEV-1'))
Your user has executed a query to return 68,000 rows - what type of user is it, a human being cannot possibly cope with that much data and it's not entirely surprising that it might take quite some time to return it.
One thing I'd check is whether you're always getting the same execution plan - Oracle's estimates here are out by a factor of about 95 (7,100 rows predicted vs. 68,500 returned) perhaps some of your variation in timing relates to plan changes.
If you check the figures you'll see about half your time came from probing the unique index, and half came from visiting the table. In general it's hard to beat Oracle's caching algorithms, but indexes are often much smaller than the tables they cover, so it's possible that your best strategy is to protect this index at the cost of the table. Rather than trying to create a KEEP cache the index, though, you MIGHT find that you get some benefit from creating a RECYCLE cache for the table, using a small percentage of the available memory - the target is to fix things so that table blocks you won't revisit don't push index blocks you will revisit from memory.
Another detail to consider is that if you are visiting the index and table completely randomly (for 68,500 locations) it's possible that you end up re-reading blocks several times in the course of the visit. If you order the intermediate result set from the from the driving table first you may find that you're walking the index and table in order and don't have to re-read any blocks. This is something only you can know, though. THe code would have to change to include an inline view with a no_merge and no_eliminate_oby hint.
Regards
Jonathan Lewis

Similar Messages

Performance question for date column

Hi All,
I have a select query which joins two tables with more than 20 million records each based on a where condition of year = 2006. Would creating an function based index on the date column help improve the performance. Please suggest any other alternatives. The query is supposed to return about 10 million records which is taking about 10 hrs right now. How can I improve the performance.
Thanks

Rob, Here is the explain plan. Any suggestions based on this information. Please refer me to any information on how to interpret this information. Thanks
| Id | Operation            | Name                      | Rows | Bytes |TempSpc| Cost | Pstart| Pstop |
|   0 | SELECT STATEMENT     |                            |   101K|    10M|       |   173K|       |       |
|   1 | HASH JOIN           |                            |   101K|    10M| 3088K|   173K|       |       |
|   2 |   PARTITION RANGE ALL|                            |       |       |       |       |     1 |     9 |
|   3 |    TABLE ACCESS FULL | Table A                    |   101K| 1888K|       | 87464 |     1 |     9 |
|   4 |   PARTITION RANGE ALL|                            |       |       |       |       |     1 |     9 |
|   5 |    TABLE ACCESS FULL | Table B                    |   156K|    13M|       | 85454 |     1 |     9 |

Performance Point Services Data Source Sql Server Table.

Hi,
I have a requirement where I need to create a matrix report which shows resource capacity and deomand fromProject Server 2010 databases which involves joining of multiple tables and views from the database.
But in Performance Point Service 2010 I am not able to write query to join multiple tables it'sjust showing one option in Data Connection to create connection to a database and specific one table not like SSRS where we can connect to database and can query
multiple tables and views to render the report as required.
So can any body suggest in case I am missing something in PPS 2010 or we cant joing or write sql query in PPS 2010
Thanks

I would join the tables in a view as Regis suggested or simply pull the information into a PowerPivot model and use that in SharePoint to connect to in PPS. If you go that route then you will be able to use a multidimensional source and can create
Analytical Reports.
http://denglishbi.wordpress.com/2011/01/03/using-powerpivot-with-performancepoint-services-pps-2010/
http://denglishbi.wordpress.com/2012/02/09/using-performancepoint-services-pps-with-powerpivot-sql-server-2012-rc0/

How to improve performance by pulling data instead of BSEG table?

Hi,
We are facing issue in which we have to pull material no for some non copa postings.
But if we use BSEG table then serious performance issues are coming up..
so are there any other tables / combination of tables that we can look for instead of BSEG?

Hi,
BSEG is Cluster table, you can only select with key fields.
if you have a select:
select belnr budat wrbtr from bseg
          into table it_bseg
           where bukrs = bukrs
               and belnr = belnr
               and gjahr = gjahr
               and bschl = 31.
it's much better to select of this way:
select belnr budat wrbtr from bseg
          into table it_bseg
           where bukrs = bukrs
               and belnr = belnr
               and gjahr = gjahr.
delete it_bseg where bschl ne '31'.
Regards,
Fernando

SELECT query performance : One big table Vs many small tables

Hello,
We are using BDB 11g with SQLITE support. I have a query about 'select' query performance when we have one huge table vs. multiple small tables.
Basically in our application, we need to run select query multiple times and today we have one huge table. Do you guys think breaking them into
multiple small tables will help ?
For test purposes we tried creating multiple tables but performance of 'select' query was more or less same. Would that be because all tables will map to only one database in backed with key/value pair and when we run lookup (select query) on small table or big table it wont make difference ?
Thanks.

Hello,
There is some information on this topic in the FAQ at:
http://www.oracle.com/technology/products/berkeley-db/faq/db_faq.html#9-63
If this does not address your question, please just let me know.
Thanks,
Sandra

In which table is the Live cache data stored?

Hi experts,
I am new APO .Can anyone let me know in which database table will the Live cache data be stored?
Thanks in advance
regards
Ashwin.

Hello Ashwin,
the idea of the liveCache is to haev data permanently and quickly available without having to read/write from/to database. Therefore, the liveCache is <b>NOT</b> a physical database, it is a program in C++ that <i>simulates</i> a database and holds the data in the memory.
However, it does need so called liveCache-anchors which are saved on the database, and which are similar to pointers.
However, you can extract data from liveCache using BADIs or by creating a datasource on a planning area (for DP and SNP), manipulation can also be done only by BADIs or sophisticated programming (which basically uses RFCs).
I hope this answers your question.
Regards,
Klaus

How can we improve the performance while fetching data from RESB table.

Hi All,
Can any bosy suggest me the right way to improve the performance while fetching data from RESB table. Below is the select statement.
SELECT aufnr posnr roms1 roanz
    INTO (itab-aufnr, itab-pposnr, itab-roms1, itab-roanz)
    FROM resb
    WHERE kdauf = p_vbeln
    AND   ablad = itab-sposnr+2.
Here I am using 'KDAUF' & 'ABLAD' in condition. Can we use secondary index for improving the performance in this case.
Regards,
Himanshu

Hi ,
Declare intenal table with only those four fields.
and try the beloe code....
SELECT aufnr posnr roms1 roanz
INTO table itab
FROM resb
WHERE kdauf = p_vbeln
AND ablad = itab-sposnr+2.
yes, you can also use secondary index for improving the performance in this case.
Regards,
Anand .
Reward if it is useful....

Question about inputting data to a pivot table

Hi,
I have 2 questions about using the ADF pivot table component (I would like it for data input).
1. Is it possible to paste into multiple cells (i.e. using cut in Excel for example, and then pasting the cells into the pivot table) ?
2. What is the recommended approach for the case where there is no data values (no rows on the database table)? For example, if I have regions, products, measures and time periods and then want to be able to select some regions and some products to enter new sales figures - and then create the rows on the database with save button, is there a recommended way to do this?
(Jdeveloper version: 11.1.2.0.0)

Anybody able to help with this?

Analyse big data in Excel? Why the dynamic tables doesn't take all the data from the source table.

Hi,
I'm doing a internship in a production line.
My job is to recover production data (input data) and test data (output data) using various types of software (excel, BusinessObject sap, etc).
To this day, I have recovered hundreds of production data, and have also organized in excel but I need to analyze and plot them.
I would like to know who can give me an idea of how I could plot as much data and analysis.
Now i trying to use dynamic charts and plot some data but I did not get acceptable answers.
How could I compare, analyze and graph for example:
Five columns of production (input) with five (5) columns tested (data output).
After graphing.
Someone can give me a technique to analyze data? ie I compare column by column?
or some other technique? as a conglomerate could analyze data?
o give you an idea of the contect, now I perform an internship in a manufacturing turbines.
My job is to analyze the input data (production) and to estimate the possible behavior of the turbines in the tests.
As I said, use dynamic tables in excel, but i have not idea why the dynamic tables doesn't take all the data from the source table.
I appreciate your advice
Thanks

You can declare as PT source whole Columns [$A:$E], without rows number.
Then You'll have all actually data.
Oskar Shon, Office System MVP - www.VBATools.pl
if Helpful; Answer when a problem solved

BPM question - Delete all entries in a table and Insert data

Hello,
I'm new to PI BPM and have this scenario to implement.
I'm supposed to delete all entries in a table and then insert data to the same table.
Is BPM necessary? If it is, what is the best practice to implement my scenario?
I want to make sure that all entries were successfully deleted before insert data.
How do I check if all entries in the table were deleted successfully before
initiating insert.
Thank you.
-Won

Hi Won,
Yes, this is possible without BPM.
You have to create 2 separate mappings, each for deleting & inserting the records.
You will have 2 different Interface mappings as well.
But in ID, create a single interface determination.
In that, configure Deletion interface mapping first and in the second row, configure Insertion interface mapping.
Please make sure that the parameter 'Mainain Order At runtime' is checked in interface determination.
-Supriya.

How to select the data efficiently from the table

hi every one,
i need some help in selecting data from FAGLFLEXA table.i have to select many amounts from different group of G/L accounts
(groups are predefined here which contains a set of g/L account no.).
if i select every time for each group then it will be a performance issue, in order to avoid it what should i do, can any one suggest me a method or a smaple query so that i can perform the task efficiently.

Hi ,
1.select and keep the data in internal table
2.avoid select inside loop ..endloop.
3.try to use for all entries
check the below details
Hi Praveen,
Performance Notes
1.Keep the Result Set Small
You should aim to keep the result set small. This reduces both the amount of memory used in the database system and the network load when transferring data to the application server. To reduce the size of your result sets, use the WHERE and HAVING clauses.
Using the WHERE Clause
Whenever you access a database table, you should use a WHERE clause in the corresponding Open SQL statement. Even if a program containing a SELECT statement with no WHERE clause performs well in tests, it may slow down rapidly in your production system, where the data volume increases daily. You should only dispense with the WHERE clause in exceptional cases where you really need the entire contents of the database table every time the statement is executed.
When you use the WHERE clause, the database system optimizes the access and only transfers the required data. You should never transfer unwanted data to the application server and then filter it using ABAP statements.
Using the HAVING Clause
After selecting the required lines in the WHERE clause, the system then processes the GROUP BY clause, if one exists, and summarizes the database lines selected. The HAVING clause allows you to restrict the grouped lines, and in particular, the aggregate expressions, by applying further conditions.
Effect
If you use the WHERE and HAVING clauses correctly:
There are no more physical I/Os in the database than necessary
No unwanted data is stored in the database cache (it could otherwise displace data that is actually required)
The CPU usage of the database host is minimize
The network load is reduced, since only the data that is required by the application is transferred to the application server.
Minimize the Amount of Data Transferred
Data is transferred between the database system and the application server in blocks. Each block is up to 32 KB in size (the precise size depends on your network communication hardware). Administration information is transported in the blocks as well as the data.
To minimize the network load, you should transfer as few blocks as possible. Open SQL allows you to do this as follows:
Restrict the Number of Lines
If you only want to read a certain number of lines in a SELECT statement, use the UP TO <n> ROWS addition in the FROM clause. This tells the database system only to transfer <n> lines back to the application server. This is more efficient than transferring more lines than necessary back to the application server and then discarding them in your ABAP program.
If you expect your WHERE clause to return a large number of duplicate entries, you can use the DISTINCT addition in the SELECT clause.
Restrict the Number of Columns
You should only read the columns from a database table that you actually need in the program. To do this, list the columns in the SELECT clause. Note here that the INTO CORRESPONDING FIELDS addition in the INTO clause is only efficient with large volumes of data, otherwise the runtime required to compare the names is too great. For small amounts of data, use a list of variables in the INTO clause.
Do not use * to select all columns unless you really need them. However, if you list individual columns, you may have to adjust the program if the structure of the database table is changed in the ABAP Dictionary. If you specify the database table dynamically, you must always read all of its columns.
Use Aggregate Functions
If you only want to use data for calculations, it is often more efficient to use the aggregate functions of the SELECT clause than to read the individual entries from the database and perform the calculations in the ABAP program.
Aggregate functions allow you to find out the number of values and find the sum, average, minimum, and maximum values.
Following an aggregate expression, only its result is transferred from the database.
Data Transfer when Changing Table Lines
When you use the UPDATE statement to change lines in the table, you should use the WHERE clause to specify the relevant lines, and then SET statements to change only the required columns.
When you use a work area to overwrite table lines, too much data is often transferred. Furthermore, this method requires an extra SELECT statement to fill the work area. Minimize the Number of Data Transfers
In every Open SQL statement, data is transferred between the application server and the database system. Furthermore, the database system has to construct or reopen the appropriate administration data for each database access. You can therefore minimize the load on the network and the database system by minimizing the number of times you access the database.
Multiple Operations Instead of Single Operations
When you change data using INSERT, UPDATE, and DELETE, use internal tables instead of single entries. If you read data using SELECT, it is worth using multiple operations if you want to process the data more than once, other wise, a simple select loop is more efficient.
Avoid Repeated Access
As a rule you should read a given set of data once only in your program, and using a single access. Avoid accessing the same data more than once (for example, SELECT before an UPDATE).
Avoid Nested SELECT Loops
A simple SELECT loop is a single database access whose result is passed to the ABAP program line by line. Nested SELECT loops mean that the number of accesses in the inner loop is multiplied by the number of accesses in the outer loop. You should therefore only use nested SELECT loops if the selection in the outer loop contains very few lines.
However, using combinations of data from different database tables is more the rule than the exception in the relational data model. You can use the following techniques to avoid nested SELECT statements:
ABAP Dictionary Views
You can define joins between database tables statically and systemwide as views in the ABAP Dictionary. ABAP Dictionary views can be used by all ABAP programs. One of their advantages is that fields that are common to both tables (join fields) are only transferred once from the database to the application server.
Views in the ABAP Dictionary are implemented as inner joins. If the inner table contains no lines that correspond to lines in the outer table, no data is transferred. This is not always the desired result. For example, when you read data from a text table, you want to include lines in the selection even if the corresponding text does not exist in the required language. If you want to include all of the data from the outer table, you can program a left outer join in ABAP.
The links between the tables in the view are created and optimized by the database system. Like database tables, you can buffer views on the application server. The same buffering rules apply to views as to tables. In other words, it is most appropriate for views that you use mostly to read data. This reduces the network load and the amount of physical I/O in the database.
Joins in the FROM Clause
You can read data from more than one database table in a single SELECT statement by using inner or left outer joins in the FROM clause.
The disadvantage of using joins is that redundant data is read from the hierarchically-superior table if there is a 1:N relationship between the outer and inner tables. This can considerably increase the amount of data transferred from the database to the application server. Therefore, when you program a join, you should ensure that the SELECT clause contains a list of only the columns that you really need. Furthermore, joins bypass the table buffer and read directly from the database. For this reason, you should use an ABAP Dictionary view instead of a join if you only want to read the data.
The runtime of a join statement is heavily dependent on the database optimizer, especially when it contains more than two database tables. However, joins are nearly always quicker than using nested SELECT statements.
Subqueries in the WHERE and HAVING Clauses
Another way of accessing more than one database table in the same Open SQL statement is to use subqueries in the WHERE or HAVING clause. The data from a subquery is not transferred to the application server. Instead, it is used to evaluate conditions in the database system. This is a simple and effective way of programming complex database operations.
Using Internal Tables
It is also possible to avoid nested SELECT loops by placing the selection from the outer loop in an internal table and then running the inner selection once only using the FOR ALL ENTRIES addition. This technique stems from the time before joins were allowed in the FROM clause. On the other hand, it does prevent redundant data from being transferred from the database.
Using a Cursor to Read Data
A further method is to decouple the INTO clause from the SELECT statement by opening a cursor using OPEN CURSOR and reading data line by line using FETCH NEXT CURSOR. You must open a new cursor for each nested loop. In this case, you must ensure yourself that the correct lines are read from the database tables in the correct order. This usually requires a foreign key relationship between the database tables, and that they are sorted by the foreign key. Minimize the Search Overhead
You minimize the size of the result set by using the WHERE and HAVING clauses. To increase the efficiency of these clauses, you should formulate them to fit with the database table indexes.
Database Indexes
Indexes speed up data selection from the database. They consist of selected fields of a table, of which a copy is then made in sorted order. If you specify the index fields correctly in a condition in the WHERE or HAVING clause, the system only searches part of the index (index range scan).
The primary index is always created automatically in the R/3 System. It consists of the primary key fields of the database table. This means that for each combination of fields in the index, there is a maximum of one line in the table. This kind of index is also known as UNIQUE.
If you cannot use the primary index to determine the result set because, for example, none of the primary index fields occur in the WHERE or HAVING clause, the system searches through the entire table (full table scan). For this case, you can create secondary indexes, which can restrict the number of table entries searched to form the result set.
You specify the fields of secondary indexes using the ABAP Dictionary. You can also determine whether the index is unique or not. However, you should not create secondary indexes to cover all possible combinations of fields.
Only create one if you select data by fields that are not contained in another index, and the performance is very poor. Furthermore, you should only create secondary indexes for database tables from which you mainly read, since indexes have to be updated each time the database table is changed. As a rule, secondary indexes should not contain more than four fields, and you should not have more than five indexes for a single database table.
If a table has more than five indexes, you run the risk of the optimizer choosing the wrong one for a particular operation. For this reason, you should avoid indexes with overlapping contents.
Secondary indexes should contain columns that you use frequently in a selection, and that are as highly selective as possible. The fewer table entries that can be selected by a certain column, the higher that columns selectivity. Place the most selective fields at the beginning of the index. Your secondary index should be so selective that each index entry corresponds to at most five percent of the table entries. If this is not the case, it is not worth creating the index. You should also avoid creating indexes for fields that are not always filled, where their value is initial for most entries in the table.
If all of the columns in the SELECT clause are contained in the index, the system does not have to search the actual table data after reading from the index. If you have a SELECT clause with very few columns, you can improve performance dramatically by including these columns in a secondary index.
Formulating Conditions for Indexes
You should bear in mind the following when formulating conditions for the WHERE and HAVING clauses so that the system can use a database index and does not have to use a full table scan.
Check for Equality and Link Using AND
The database index search is particularly efficient if you check all index fields for equality (= or EQ) and link the expressions using AND.
Use Positive Conditions
The database system only supports queries that describe the result in positive terms, for example, EQ or LIKE. It does not support negative expressions like NE or NOT LIKE.
If possible, avoid using the NOT operator in the WHERE clause, because it is not supported by database indexes; invert the logical expression instead.
Using OR
The optimizer usually stops working when an OR expression occurs in the condition. This means that the columns checked using OR are not included in the index search. An exception to this are OR expressions at the outside of conditions. You should try to reformulate conditions that apply OR expressions to columns relevant to the index, for example, into an IN condition.
Using Part of the Index
If you construct an index from several columns, the system can still use it even if you only specify a few of the columns in a condition. However, in this case, the sequence of the columns in the index is important. A column can only be used in the index search if all of the columns before it in the index definition have also been specified in the condition.
Checking for Null Values
The IS NULL condition can cause problems with indexes. Some database systems do not store null values in the index structure. Consequently, this field cannot be used in the index.
Avoid Complex Conditions
Avoid complex conditions, since the statements have to be broken down into their individual components by the database system.
Reduce the Database Load
Unlike application servers and presentation servers, there is only one database server in your system. You should therefore aim to reduce the database load as much as possible. You can use the following methods:
Buffer Tables on the Application Server
You can considerably reduce the time required to access data by buffering it in the application server table buffer. Reading a single entry from table T001 can take between 8 and 600 milliseconds, while reading it from the table buffer takes 0.2 - 1 milliseconds.
Whether a table can be buffered or not depends its technical attributes in the ABAP Dictionary. There are three buffering types:
Resident buffering (100%) The first time the table is accessed, its entire contents are loaded in the table buffer.
Generic buffering In this case, you need to specify a generic key (some of the key fields) in the technical settings of the table in the ABAP Dictionary. The table contents are then divided into generic areas. When you access data with one of the generic keys, the whole generic area is loaded into the table buffer. Client-specific tables are often buffered generically by client.
Partial buffering (single entry) Only single entries are read from the database and stored in the table buffer.
When you read from buffered tables, the following happens:
1. An ABAP program requests data from a buffered table.
2. The ABAP processor interprets the Open SQL statement. If the table is defined as a buffered table in the ABAP Dictionary, the ABAP processor checks in the local buffer on the application server to see if the table (or part of it) has already been buffered.
3. If the table has not yet been buffered, the request is passed on to the database. If the data exists in the buffer, it is sent to the program.
4. The database server passes the data to the application server, which places it in the table buffer.
5. The data is passed to the program.
When you change a buffered table, the following happens:
1. The database table is changed and the buffer on the application server is updated. The database interface logs the update statement in the table DDLOG. If the system has more than one application server, the buffer on the other servers is not updated at once.
2. All application servers periodically read the contents of table DDLOG, and delete the corresponding contents from their buffers where necessary. The granularity depends on the buffering type. The table buffers in a distributed system are generally synchronized every 60 seconds (parameter: rsdisp/bufreftime).
3. Within this period, users on non-synchronized application servers will read old data. The data is not recognized as obsolete until the next buffer synchronization. The next time it is accessed, it is re-read from the database.
You should buffer the following types of tables:
Tables that are read very frequently
Tables that are changed very infrequently
Relatively small tables (few lines, few columns, or short columns)
Tables where delayed update is acceptable.
Once you have buffered a table, take care not to use any Open SQL statements that bypass the buffer.
The SELECT statement bypasses the buffer when you use any of the following:
The BYPASSING BUFFER addition in the FROM clause
The DISTINCT addition in the SELECT clause
Aggregate expressions in the SELECT clause
Joins in the FROM clause
The IS NULL condition in the WHERE clause
Subqueries in the WHERE clause
The ORDER BY clause
The GROUP BY clause
The FOR UPDATE addition
Furthermore, all Native SQL statements bypass the buffer.
Avoid Reading Data Repeatedly
If you avoid reading the same data repeatedly, you both reduce the number of database accesses and reduce the load on the database. Furthermore, a "dirty read" may occur with database tables other than Oracle. This means that the second time you read data from a database table, it may be different from the data read the first time. To ensure that the data in your program is consistent, you should read it once only and then store it in an internal table.
Sort Data in Your ABAP Programs
The ORDER BY clause in the SELECT statement is not necessarily optimized by the database system or executed with the correct index. This can result in increased runtime costs. You should only use ORDER BY if the database sort uses the same index with which the table is read. To find out which index the system uses, use SQL Trace in the ABAP Workbench Performance Trace. If the indexes are not the same, it is more efficient to read the data into an internal table or extract and sort it in the ABAP program using the SORT statement.
Use Logical Databases
SAP supplies logical databases for all applications. A logical database is an ABAP program that decouples Open SQL statements from application programs. They are optimized for the best possible database performance. However, it is important that you use the right logical database. The hierarchy of the data you want to read must reflect the structure of the logical database, otherwise, they can have a negative effect on performance. For example, if you want to read data from a table right at the bottom of the hierarchy of the logical database, it has to read at least the key fields of all tables above it in the hierarchy. In this case, it is more efficient to use a SELECT statement.
Work Processes
Work processes execute the individual dialog steps in R/3 applications. The next two sections describe firstly the structure of a work process, and secondly the different types of work process in the R/3 System.
Structure of a Work Process
Work processes execute the dialog steps of application programs. They are components of an application server. The following diagram shows the components of a work process:
Each work process contains two software processors and a database interface.
Screen Processor
In R/3 application programming, there is a difference between user interaction and processing logic. From a programming point of view, user interaction is controlled by screens. As well as the actual input mask, a screen also consists of flow logic. The screen flow logic controls a large part of the user interaction. The R/3 Basis system contains a special language for programming screen flow logic. The screen processor executes the screen flow logic. Via the dispatcher, it takes over the responsibility for communication between the work process and the SAPgui, calls modules in the flow logic, and ensures that the field contents are transferred from the screen to the flow logic.
ABAP Processor
The actual processing logic of an application program is written in ABAP - SAPs own programming language. The ABAP processor executes the processing logic of the application program, and communicates with the database interface. The screen processor tells the ABAP processor which module of the screen flow logic should be processed next. The following screen illustrates the interaction between the screen and the ABAP processors when an application program is running.
Database Interface
The database interface provides the following services:
Establishing and terminating connections between the work process and the database.
Access to database tables
Access to R/3 Repository objects (ABAP programs, screens and so on)
Access to catalog information (ABAP Dictionary)
Controlling transactions (commit and rollback handling)
Table buffer administration on the application server.
The following diagram shows the individual components of the database interface:
The diagram shows that there are two different ways of accessing databases: Open SQL and Native SQL.
Open SQL statements are a subset of Standard SQL that is fully integrated in ABAP. They allow you to access data irrespective of the database system that the R/3 installation is using. Open SQL consists of the Data Manipulation Language (DML) part of Standard SQL; in other words, it allows you to read (SELECT) and change (INSERT, UPDATE, DELETE) data. The tasks of the Data Definition Language (DDL) and Data Control Language (DCL) parts of Standard SQL are performed in the R/3 System by the ABAP Dictionary and the authorization system. These provide a unified range of functions, irrespective of database, and also contain functions beyond those offered by the various database systems.
Open SQL also goes beyond Standard SQL to provide statements that, in conjunction with other ABAP constructions, can simplify or speed up database access. It also allows you to buffer certain tables on the application server, saving excessive database access. In this case, the database interface is responsible for comparing the buffer with the database. Buffers are partly stored in the working memory of the current work process, and partly in the shared memory for all work processes on an application server. Where an R/3 System is distributed across more than one application server, the data in the various buffers is synchronized at set intervals by the buffer management. When buffering the database, you must remember that data in the buffer is not always up to date. For this reason, you should only use the buffer for data which does not often change.
Native SQL is only loosely integrated into ABAP, and allows access to all of the functions contained in the programming interface of the respective database system. Unlike Open SQL statements, Native SQL statements are not checked and converted, but instead are sent directly to the database system. Programs that use Native SQL are specific to the database system for which they were written. R/3 applications contain as little Native SQL as possible. In fact, it is only used in a few Basis components (for example, to create or change table definitions in the ABAP Dictionary).
The database-dependent layer in the diagram serves to hide the differences between database systems from the rest of the database interface. You choose the appropriate layer when you install the Basis system. Thanks to the standardization of SQL, the differences in the syntax of statements are very slight. However, the semantics and behavior of the statements have not been fully standardized, and the differences in these areas can be greater. When you use Native SQL, the function of the database-dependent layer is minimal.
Types of Work Process
Although all work processes contain the components described above, they can still be divided into different types. The type of a work process determines the kind of task for which it is responsible in the application server. It does not specify a particular set of technical attributes. The individual tasks are distributed to the work processes by the dispatcher.
Before you start your R/3 System, you determine how many work processes it will have, and what their types will be. The dispatcher starts the work processes and only assigns them tasks that correspond to their type. This means that you can distribute work process types to optimize the use of the resources on your application servers.
The following diagram shows again the structure of an application server, but this time, includes the various possible work process types:
The various work processes are described briefly below. Other parts of this documentation describe the individual components of the application server and the R/3 System in more detail.
Dialog Work Process
Dialog work processes deal with requests from an active user to execute dialog steps.
Update Work Process
Update work processes execute database update requests. Update requests are part of an SAP LUW that bundle the database operations resulting from the dialog in a database LUW for processing in the background.
Background Work Process
Background work processes process programs that can be executed without user interaction (background jobs).
Enqueue Work Process
The enqueue work process administers a lock table in the shared memory area. The lock table contains the logical database locks for the R/3 System and is an important part of the SAP LUW concept. In an R/3 System, you may only have one lock table. You may therefore also only have one application server with enqueue work processes.
Spool Work Process
The spool work process passes sequential datasets to a printer or to optical archiving. Each application server may contain several spool work process.
The services offered by an application server are determined by the types of its work processes. One application server may, of course, have more than one function. For example, it may be both a dialog server and the enqueue server, if it has several dialog work processes and an enqueue work process.
You can use the system administration functions to switch a work process between dialog and background modes while the system is still running. This allows you, for example, to switch an R/3 System between day and night operation, where you have more dialog than background work processes during the day, and the other way around during the night.
ABAP Application Server
R/3 programs run on application servers. They are an important component of the R/3 System. The following sections describe application servers in more detail.
Structure of an ABAP Application Server
The application layer of an R/3 System is made up of the application servers and the message server. Application programs in an R/3 System are run on application servers. The application servers communicate with the presentation components, the database, and also with each other, using the message server.
The following diagram shows the structure of an application server:
The individual components are:
Work Processes
An application server contains work processes, which are components that can run an application. Work processes are components that are able to execute an application (that is, one dialog step each). Each work process is linked to a memory area containing the context of the application being run. The context contains the current data for the application program. This needs to be available in each dialog step. Further information about the different types of work process is contained later on in this documentation.
Dispatcher
Each application server contains a dispatcher. The dispatcher is the link between the work processes and the users logged onto the application server. Its task is to receive requests for dialog steps from the SAP GUI and direct them to a free work process. In the same way, it directs screen output resulting from the dialog step back to the appropriate user.
Gateway
Each application server contains a gateway. This is the interface for the R/3 communication protocols (RFC, CPI/C). It can communicate with other application servers in the same R/3 System, with other R/3 Systems, with R/2 Systems, or with non-SAP systems.
The application server structure as described here aids the performance and scalability of the entire R/3 System. The fixed number of work processes and dispatching of dialog steps leads to optimal memory use, since it means that certain components and the memory areas of a work process are application-independent and reusable. The fact that the individual work processes work independently makes them suitable for a multi-processor architecture. The methods used in the dispatcher to distribute tasks to work processes are discussed more closely in the section Dispatching Dialog Steps.
Shared Memory
All of the work processes on an application server use a common main memory area called shared memory to save contexts or to buffer constant data locally.
The resources that all work processes use (such as programs and table contents) are contained in shared memory. Memory management in the R/3 System ensures that the work processes always address the correct context, that is the data relevant to the current state of the program that is running. A mapping process projects the required context for a dialog step from shared memory into the address of the relevant work process. This reduces the actual copying to a minimum.
Local buffering of data in the shared memory of the application server reduces the number of database reads required. This reduces access times for application programs considerably. For optimal use of the buffer, you can concentrate individual applications (financial accounting, logistics, human resources) into separate application server groups.
Database Connection
When you start up an R/3 System, each application server registers its work processes with the database layer, and receives a single dedicated channel for each. While the system is running, each work process is a user (client) of the database system (server). You cannot change the work process registration while the system is running. Neither can you reassign a database channel from one work process to another. For this reason, a work process can only make database changes within a single database logical unit of work (LUW). A database LUW is an inseparable sequence of database operations. This has important consequences for the programming model explained below.
Dispatching Dialog Steps
The number of users logged onto an application server is often many times greater than the number of available work processes. Furthermore, it is not restricted by the R/3 system architecture. Furthermore, each user can run several applications at once. The dispatcher has the important task of distributing all dialog steps among the work processes on the application server.
The following diagram is an example of how this might happen:
1. The dispatcher receives the request to execute a dialog step from user 1 and directs it to work process 1, which happens to be free. The work process addresses the context of the application program (in shared memory) and executes the dialog step. It then becomes free again.
2. The dispatcher receives the request to execute a dialog step from user 2 and directs it to work process 1, which is now free again. The work process executes the dialog step as in step 1.
3. While work process 1 is still working, the dispatcher receives a further request from user 1 and directs it to work process 2, which is free.
4. After work processes 1 and 2 have finished processing their dialog steps, the dispatcher receives another request from user 1 and directs it to work process 1, which is free again.
5. While work process 1 is still working, the dispatcher receives a further request from user 2 and directs it to work process 2, which is free.
From this example, we can see that:
A dialog step from a program is assigned to a single work process for execution.
The individual dialog steps of a program can be executed on different work processes, and the program context must be addressed for each new work process.
A work process can execute dialog steps of different programs from different users.
The example does not show that the dispatcher tries to distribute the requests to the work processes such that the same work process is used as often as possible for the successive dialog steps in an application. This is useful, since it saves the program context having to be addressed each time a dialog step is executed.
Dispatching and the Programming Model
The separation of application and presentation layer made it necessary to split up application programs into dialog steps. This, and the fact that dialog steps are dispatched to individual work processes, has had important consequences for the programming model.
As mentioned above, a work process can only make database changes within a single database logical unit of work (LUW). A database LUW is an inseparable sequence of database operations. The contents of the database must be consistent at its beginning and end. The beginning and end of a database LUW are defined by a commit command to the database system (database commit). During a database LUW, that is, between two database commits, the database system itself ensures consistency within the database. In other words, it takes over tasks such as locking database entries while they are being edited, or restoring the old data (rollback) if a step terminates in an error.
A typical SAP application program extends over several screens and the corresponding dialog steps. The user requests database changes on the individual screens that should lead to the database being consistent once the screens have all been processed. However, the individual dialog steps run on different work processes, and a single work process can process dialog steps from other applications. It is clear that two or more independent applications whose dialog steps happen to be processed on the same work process cannot be allowed to work with the same database LUW.
Consequently, a work process must open a separate database LUW for each dialog step. The work process sends a commit command (database commit) to the database at the end of each dialog step in which it makes database changes. These commit commands are called implicit database commits, since they are not explicitly written into the application program.
These implicit database commits mean that a database LUW can be kept open for a maximum of one dialog step. This leads to a considerable reduction in database load, serialization, and deadlocks, and enables a large number of users to use the same system.
However, the question now arises of how this method (1 dialog step = 1 database LUW) can be reconciled with the demand to make commits and rollbacks dependent on the logical flow of the application program instead of the technical distribution of dialog steps. Database update requests that depend on one another form logical units in the program that extend over more than one dialog step. The database changes associated with these logical units must be executed together and must also be able to be undone together.
The SAP programming model contains a series of bundling techniques that allow you to group database updates together in logical units. The section of an R/3 application program that bundles a set of logically-associated database operations is called an SAP LUW. Unlike a database LUW, a SAP LUW includes all of the dialog steps in a logical unit, including the database update.
Happy Reading...
shibu

Retrieve data from a large table from ORACLE 10g

I am working with a Microsoft Visual Studio Project that requires to retrieve data from a large table from Oracle 10g database and export the data into the hard drive.
The problem here is that I am not able to connect to the database directly because of license issue but I can use a third party API to retrieve data from the database. This API has sufficient previllege/license permission on to the database to perform retrieval of data. So, I am not able to use DTS/SSIS or other tool to import data from the database directly connecting to it.
Here my approach is...first retrieve the data using the API into a .net DataTable and then dump the records from it into the hard drive in a specific format (might be in Excel file/ another SQL server database).
When I try to retrieve the data from a large table having over 13 lacs records (3-4 GB) in a data table using the visual studio project, I get an Out of memory exception.
But is there any better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?
Any help on this problem will be highly appriciated.
Thanks in advance...
-Jahedur Rahman
Edited by: Jahedur on May 16, 2010 11:42 PM

Girish...Thanks for your reply...But I am sorry for the confusions. Let me explain that...
1."export the data into another media into the hard drive."
What does it mean by this line i.e. another media into hard drive???
ANS: Sorry...I just want to write the data in a file or in a table in SQL server database.
2."I am not able to connect to the database directly because of license issue"
huh?? I never heard this question that a user is not able to connect the db because of license. What error / message you are getting?
ANS: My company uses a 3rd party application that uses ORACLE 10g. And my compnay is licensed to use the 3rd party application (APP+Database is a package) and did not purchased ORACLE license to use directly. So I will not connect to the database directly.
3.I am not sure which API is you are talking about, but i am running an application of the visual studio data grid or similar kind of controls; in which i can select (select query) as many rows as i needed; no issue.
ANS: This API is provided by the 3rd party application vendor. I can pass a query to it and it returns a datatable.
4."better way to retrieve the records chunk by chunk and do the export without loosing the state of the data in the table?"
ANS: As I get a system error (out of memory) when I select all rows in a datatable at a time, I wanted to retrieve the data in multiple phases.
E.g: 1 to 20,000 records in 1st phase
20,001 to 40,000 records in 2nd phase
40,001 to ...... records in 3nd phase
and so on...
Please let me know if this does not clarify your confusions... :)
Thanks...
-Jahedur Rahman
Edited by: user13114507 on May 12, 2010 11:28 PM

Neen help with date range searches for Table Sources

Hi all,
I need help, please. I am trying to satisfy a Level 1 client requirement for the ability to search for records in crawled table sources by a date and/or date range. I have performed the following steps, and did not get accurate results from Advanced searching for date. Please help me understand what I am doing wrong, and/or if there is a way to define a date search attribute without creating a LOV for a date column. (My tables have 500,00 rows.)
I am using SES 10.1.8.3 on Windows 32.
My Oracle 10g Spatial Table is called REPORTS and this table has the following columns:
TRACKNUM Varchar2
TITLE Varchar2
SUMMARY CLOB
SYMBOLCODE Varchar2
Timestamp Date
OBSDATE Date
GEOM SDO_GEOMETRY
I set up the REPORTS table source in SES, using TRACKNUM as the Primary Key (unique and not null), and SUMMARY as the CONTENT Column. In the Table Column Mappings I defined TITLE as String and TITLE.
Under Global Settings > Search Attributes I defined a new Search Attribute (type Date) called DATE OCCURRED (DD-MON-YY).
Went back to REPORTS source previously defined and added a new Table Column Mapping - mapping OBSDATE to the newly defined DATE OCCURRED (DD-MON-YY) search attribute.
I then modified the Schedule for the REPORTS source Crawler Policy to “Process All Documents”.
Schedule crawls and indexes entire REPORTS table.
In SES Advanced Search page, I enter my search keyword, select Specific Source Group as REPORTS, select All Match, and used the pick list to select the DATE OCCURRED (DD-MON-YY) Attribute Name, operator of Greater than equal, and entered the Value 01-JAN-07. Then the second attribute name of DATE_OCCURRED (DD-MON-YY), less than equals, 10-JAN-07.
Search results gave me 38,000 documents, and the first 25 I looked at had dates NOT within the 01-JAN-07 / 10-JAN-07 range. (e.g. OBSDATE= 24-MAR-07, 22-SEP-), 02-FEB-08, etc.)
And, none of the results I opened had ANY dates within the SUMMARY CLOB…in case that’s what was being found in the search.
Can someone help me figure out how to allow my client to search for specific dated records in a db table using a single column for the date? This is a major requirement and they are anxiously awaiting my solution.
Thanks, in advance….

raford,
Thanks very much for your reply. However, from what I've read in the SES Admin Document is that (I think) the date format DD/MM/YYYY pertains only to searches on "file system" sources (e.g. Word, Excel, Powerpoint, PDF, etc.). We have 3 file system sources among our 25 total sources. The remaining 22 sources are all TABLE or DATABASE sources. The DBA here has done a great job getting the data standardized using the typical/default Oracle DATE type format in our TABLE sources (DD-MON-YY). Our tables have anywhere from 1500 rows to 2 million rows.
I tested your theory that the dates we are entering are being changed to Strings behind the scenes and on the Advanced Page, searched for results using OBSDATE equals 01/02/2007 in an attempt to find data that I know for certain to be in the mapped OBSDATE table column as 01-FEB-07. My result set contained data that had an OBSDATE of 03-MAR-07 and none containing 01-FEB-07.
Here is the big issue...in order for my client to fulfill his primary mission, one of the top 5 requirements is that he/she be able to find specific table rows that are contain a specific date or range of dates.
thanks very much!

Need help with date range searches for Table Sources in SES

Hi all,
I need help, please. I am trying to satisfy a Level 1 client requirement for the ability to search for records in crawled table sources by a date and/or date range. I have performed the following steps, and did not get accurate results from Advanced searching for date. Please help me understand what I am doing wrong, and/or if there is a way to define a date search attribute without creating a LOV for a date column. (My tables have 500,00 rows.)
I am using SES 10.1.8.3 on Windows 32.
My Oracle 10g Spatial Table is called REPORTS and this table has the following columns:
TRACKNUM Varchar2
TITLE Varchar2
SUMMARY CLOB
SYMBOLCODE Varchar2
Timestamp Date
OBSDATE Date
GEOM SDO_GEOMETRY
I set up the REPORTS table source in SES, using TRACKNUM as the Primary Key (unique and not null), and SUMMARY as the CONTENT Column. In the Table Column Mappings I defined TITLE as String and TITLE.
Under Global Settings > Search Attributes I defined a new Search Attribute (type Date) called DATE OCCURRED (DD-MON-YY).
Went back to REPORTS source previously defined and added a new Table Column Mapping - mapping OBSDATE to the newly defined DATE OCCURRED (DD-MON-YY) search attribute.
I then modified the Schedule for the REPORTS source Crawler Policy to “Process All Documents”.
Schedule crawls and indexes entire REPORTS table.
In SES Advanced Search page, I enter my search keyword, select Specific Source Group as REPORTS, select All Match, and used the pick list to select the DATE OCCURRED (DD-MON-YY) Attribute Name, operator of Greater than equal, and entered the Value 01-JAN-07. Then the second attribute name of DATE_OCCURRED (DD-MON-YY), less than equals, 10-JAN-07.
Search results gave me 38,000 documents, and the first 25 I looked at had dates NOT within the 01-JAN-07 / 10-JAN-07 range. (e.g. OBSDATE= 10-MAR-07, 22-SEP-07, 02-FEB-08, etc.)
And, none of the results I opened had ANY dates within the SUMMARY CLOB…in case that’s what was being found in the search.
Can someone help me figure out how to allow my client to search for specific dated records in a db table using a single column for the date? This is a major requirement and they are anxiously awaiting my solution.
Thanks very much, in advance….

raford,
Thanks very much for your reply. However, from what I've read in the SES Admin Document is that (I think) the date format DD/MM/YYYY pertains only to searches on "file system" sources (e.g. Word, Excel, Powerpoint, PDF, etc.). We have 3 file system sources among our 25 total sources. The remaining 22 sources are all TABLE or DATABASE sources. The DBA here has done a great job getting the data standardized using the typical/default Oracle DATE type format in our TABLE sources (DD-MON-YY). Our tables have anywhere from 1500 rows to 2 million rows.
I tested your theory that the dates we are entering are being changed to Strings behind the scenes and on the Advanced Page, searched for results using OBSDATE equals 01/02/2007 in an attempt to find data that I know for certain to be in the mapped OBSDATE table column as 01-FEB-07. My result set contained data that had an OBSDATE of 03-MAR-07 and none containing 01-FEB-07.
Here is the big issue...in order for my client to fulfill his primary mission, one of the top 5 requirements is that he/she be able to find specific table rows that are contain a specific date or range of dates.
thanks very much!

Update data automatically in fact table in Data Warehouse

Hi,
I'm working on the creation of a data warehouse that include different data source like SQL Server performance (more than one), Active Directory users, Server performance (more than one), Exchange server mailboxes. The problem is that performance data change
frequently (like CPU and Memory), so my question is how to update data in fact table every 5 seconds automatically with SSIS.
Thank you for any advice

I'm assuming you have already figured out how to capture the data e.g. Powershell, extended events, MDW etc. and just need to know what dimensions or fact tables do you need.
You need to decide how often you are going to capture this data and based on that you will have dimensions with appropriate grain. Don't try to cram everything in the same fact table if it not of the same granularity. Also, separate process usually
have separate fact tables.
In addition to the Date dimension, you will need a Time dimension with a grain of 1 second (or maybe 5 seconds if that is when you get your data) then run the SSIS every 5 seconds to capture and append that data in the fact table.
- Aalamjeet Rangi | (Blog)

Performance question - Caching data of a big table

Similar Messages

Maybe you are looking for