Query Performance using regexp_like
I have a question regarding the performance of using regular expression in a sql query.
Let me explain what I am trying to achieve.
I have a table, EXPRESSIONS, containing regular expression and its ID.
I have a table, DATA, containing data that I want to map to a possible regular expression ID.
Data is being added daily into the DATA table. The rows are marked initially with an invalid expression ID.
A job runs regularly that selects all the unmapped rows at that point in time in the DATA table and try to map each of them to a regular expression.
The problem is that as the number of unmapped rows grows, the mapping job does not scale good enough. This is due to the fact that, in the worst case, a data row may need to iterate through all the regular expressions in EXPRESSIONS table.
The number of expressions would be much much less about 100 compared to 2 million of possible data rows.
What are my options to speed up the process?
Is it possible to pre-compile the regular expressions?
Thanks for you help.
Your question "Is it possible to pre-compile the regular expressions?" seems to be the answer. You may calculate and store hash values in both EXPRESSIONS and DATA table ,it would make the matching faster.
Similar Messages
-
How to improve query performance using infoset
I create one infoset that including 4 char.and 3 DSO which all are time-dependent.When query run, system show very poor perfomance, sometimes no data show in BEX anayzer. In this case I have to close BEX analyzer at first and then open it again, after that it show real results. It seems very strange. Does anybody has experience on infoset performance improvement. pls info, thanks!
Hi
As info set itself doesn't have any data so it improves Performance
also go through the below tips.
Find the query Run-time
where to find the query Run-time ?
557870 'FAQ BW Query Performance'
130696 - Performance trace in BW
This info may be helpful.
General tips
Using aggregates and compression.
Using less and complex cell definitions if possible.
1. Avoid using too many nav. attr
2. Avoid RKF and CKF
3. Many chars in row.
By using T-codes ST03 or ST03N
Go to transaction ST03 > switch to expert mode > from left side menu > and there in system load history and distribution for a particular day > check query execution time.
Statistical Records Part 4: How to read ST03N datasets from DB in NW2004
How to read ST03N datasets from DB
Try table rsddstats to get the statistics
Using cache memory will decrease the loading time of the report.
Run reporting agent at night and sending results to email. This will ensure use of OLAP cache. So later report execution will retrieve the result faster from the OLAP cache.
Also try
1. Use different parameters in ST03 to see the two important parameters aggregation ratio and records transferred to F/E to DB selected.
2. Use the program SAP_INFOCUBE_DESIGNS (Performance of BW infocubes) to see the aggregation ratio for the cube. If the cube does not appear in the list of this report, try to run RSRV checks on the cube and aggregates.
Go to SE38 > Run the program SAP_INFOCUBE_DESIGNS
It will shown dimension Vs Fact tables Size in percent.If you mean speed of queries on a cube as performance metric of cube,measure query runtime.
3. To check the performance of the aggregates,see the columns valuation and usage in aggregates.
Open the Aggregates...and observe VALUATION and USAGE columns.
"---" sign is the valuation of the aggregate. You can say -3 is the valuation of the aggregate design and usage. ++ means that its compression is good and access is also more (in effect, performance is good). If you check its compression ratio, it must be good. -- means the compression ratio is not so good and access is also not so good (performance is not so good).The more is the positives...more is useful the aggregate and more it satisfies the number of queries. The greater the number of minus signs, the worse the evaluation of the aggregate. The larger the number of plus signs, the better the evaluation of the aggregate.
if "-----" then it means it just an overhead. Aggregate can potentially be deleted and "+++++" means Aggregate is potentially very useful.
In valuation column,if there are more positive sign it means that the aggregate performance is good and it is useful to have this aggregate.But if it has more negative sign it means we need not better use that aggregate.
In usage column,we will come to know how far the aggregate has been used in query.
Thus we can check the performance of the aggregate.
Refer.
http://help.sap.com/saphelp_nw70/helpdata/en/b8/23813b310c4a0ee10000000a114084/content.htm
http://help.sap.com/saphelp_nw70/helpdata/en/60/f0fb411e255f24e10000000a1550b0/frameset.htm
performance ISSUE related to AGGREGATE
Note 356732 - Performance Tuning for Queries with Aggregates
Note 166433 - Options for finding aggregates (find optimal aggregates for an InfoCube)
4. Run your query in RSRT and run the query in the debug mode. Select "Display Aggregates Found" and "Do not use cache" in the debug mode. This will tell you if it hit any aggregates while running. If it does not show any aggregates, you might want to redesign your aggregates for the query.
Also your query performance can depend upon criteria and since you have given selection only on one infoprovider...just check if you are selecting huge amount of data in the report
Check for the query read mode in RSRT.(whether its A,X or H)..advisable read mode is X.
5. In BI 7 statistics need to be activated for ST03 and BI admin cockpit to work.
By implementing BW Statistics Business Content - you need to install, feed data and through ready made reports which for analysis.
http://help.sap.com/saphelp_nw70/helpdata/en/26/4bc0417951d117e10000000a155106/frameset.htm
/people/vikash.agrawal/blog/2006/04/17/query-performance-150-is-aggregates-the-way-out-for-me
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/1955ba90-0201-0010-d3aa-8b2a4ef6bbb2
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/ce7fb368-0601-0010-64ba-fadc985a1f94
http://help.sap.com/saphelp_nw04/helpdata/en/c1/0dbf65e04311d286d6006008b32e84/frameset.htm
You can go to T-Code DB20 which gives you all the performance related information like
Partitions
Databases
Schemas
Buffer Pools
Tablespaces etc
use tool RSDDK_CHECK_AGGREGATE in se38 to check for the corrupt aggregates
If aggregates contain incorrect data, you must regenerate them.
202469 - Using aggregate check tool
Note 646402 - Programs for checking aggregates (as of BW 3.0B SP15)
You can find out whether an aggregate is usefull or useless you can find out through a proccess of checking the tables RSDDSTATAGGRDEF*
Run the query in RSRT with statistics execute and come back you will get STATUID... copy this and check in the table...
This gives you exactly which infoobjects it's hitting, if any one of the object is missing it's useless aggregate.
6
Check SE11 > table RSDDAGGRDIR . You can find the last callup in the table.
Generate Report in RSRT
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/cccad390-0201-0010-5093-fd9ec8157802
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/4c0ab590-0201-0010-bd9a-8332d8b4f09c
Business Intelligence Journal Improving Query Performance in Data Warehouses
http://www.tdwi.org/Publications/BIJournal/display.aspx?ID=7891
Achieving BI Query Performance Building Business Intelligence
http://www.dmreview.com/issues/20051001/1038109-1.html
Assign points if useful
Cheers
SM -
Query performance using distinct
Greetings! We're on Oracle 8.1.7, Solaris 2.8.
I have a query that utilizes a different access path if I use the word distinct in the select from this view. Here is our query:
SELECT
DISTINCT SETID,
VENDOR_ID,
VENDOR_NAME_SHORT,
AR_NUM,
NAME1,
ADDRESS1,
ADDRESS2,
CITY
FROM
PS_VENDOR_VW
WHERE
SETID LIKE 'MNSA' AND
NAME1='FUN' ORDER BY NAME1, SETID, VENDOR_ID
The view SQL is:
SELECT /*+ FIRST_ROWS */ a.setid
, a.vendor_name_short
, a.name1
, c.address1
, c.address2
, c.city
, a.vendor_id
FROM SYSADM.ps_vendor a
SELECT /*+ INDEX_ASC(B PSAVENDOR_ADDR) */
FROM SYSADM.ps_vendor_addr b
WHERE b.effdt = (
SELECT MAX(effdt)
FROM SYSADM.ps_vendor_addr
WHERE setid = b.setid
AND vendor_id = b.vendor_id
AND address_seq_num = b.address_seq_num
AND effdt <= sysdate)) c
WHERE a.setid = c.setid (+)
AND a.vendor_id = c.vendor_id (+)
AND a.prim_addr_seq_num=c.address_seq_num (+)
This query does an index full scan on an index for ps_vendor_addr. It takes 2+ minutes to run. Now, if I remove the word distinct, it uses an index range scan and "view pushed predicate". It runs in 2 seconds.
I've tried with and without the first_rows hint in the view. If I leave off the INDEX_ASC hint then it does a full table scan of table ps_vendor_addr. It refuses to do a range scan with the hint. Can anybody tell me how I can get the 'distinct' query tuned?
2 minutes may not seem like a lot but when online users run the query many times a day, it is very frustrating.
Thanks! I hope I provided enough info.Thomas:
The different behaviours you are seeing are a result of the DISTINCT in the query. This causes a sort to be performed, and will influence the way that the CBO will execute the query. (You do know that you are using the Cost Based Optimizer because of the hints, and that you should analyze the tables and indexes?) You need to be able to re-write the view to avoid the need for the DISTINCT in the query.
Without knowing the meaning of the fields, it is really hard to say anything meaningful, but my guess is that it is the correlated sub-query that is ultimately causing the need for the DISTINCT. Is the combination of set_id,vendor_id and address_seq_num truly unique, or is the address_seq_num just a sequence.
For example in one of my databases, I have a table with INDV_ID, EFF_DT, EMPSTAT_SEQ. The empstat_seq field is just there to allow for more than one thing happening on the same day. The way we query this table is:
SELECT *
FROM empstat_t a
WHERE indv_id = :emp_id and
TO_CHAR(eff_dt,'yyyymmdd')||TO_CHAR(empstat_seq,'009') =
(SELECT MAX(TO_CHAR(eff_dt,'yyyymmdd')||TO_CHAR(empstat_seq,'009')
FROM empstat_t
WHERE a.indv_id = indv_id);Could something similar work in your case?
If not, assuming your statistics are up to date, I would also look at creating the view without hints to see what the optimizer comes up with on its own. It may be better than you think.
TTFN
John -
kindly help me on this
DECLARE
STRING1 VARCHAR2(1000) :='NOT ( VIP5';
BEGIN
if regexp_like(string1 ,'NOT \(* VIP5') then
dbms_output.put_line('string1 is correct');
dbms_output.PUT_LINE(STRING1);
end if;
END;problem is STRING1 can be
'NOT ( VIP5 '
OR can be NOT ( VIP5'ie between NOT AND VIP5 brackets will be there with spaces or without spaces
so help me for the output
S
Edited by: oraclehema on Mar 1, 2011 3:38 AMHAI
it is working
using
declare
STRING1 VARCHAR2(1000) :='NOT ( VIP5';
STRING2 VARCHAR2(1000) :='NOT ( ( ( VIP5';
begin
if REGEXP_LIKE( string2, 'NOT[[:space:]]*\([[:space:]]*\([[:space:]]*\([[:space:]]*VIP5') then
dbms_output.put_line('string2 is correct');
dbms_output.PUT_LINE(STRING2);
end if;
end;Edited by: oraclehema on Mar 1, 2011 3:52 AM -
"Improving SQL query performance using secondary indexes"
I have a very old copy of this document from 1997. I'm hoping to find newer version, if one exists, but the search facility on SDN is not working at the moment. Does anyone have a more up to date copy or link they can point me to ?
thanks,
Malcolm.HI,
check it out , may be it will help you
[http://www.stanford.edu/dept/itss/docs/oracle/10g/server.101/b10743/schema.htm]
[http://teradata.uark.edu/research/wang/indexes.html]
[http://www.geekinterview.com/question_details/33720] -
Hi Xperts,
i know that we can test the query performance using the RSRT Debug mode.There so many options in RSRT i am so confused.
Can any body teach me or send me docs to test query performance suing RSRT.
[email protected]hi Nrupal,
check
http://help.sap.com/saphelp_nw70/helpdata/en/a0/2a183d30805c59e10000000a114084/frameset.htm
pdf BEx Monitor(RSRT) is available in SDN ...
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/2d589190-0201-0010-f19a-c74465ce6e0f
also review query performance oss and docs
https://forums.sdn.sap.com/click.jspa?messageID=1182708
hope this helps. -
Step by step usage of RSRT for Query performance
Hi,
Can I get any step by step document, how to test the query performance using RSRT?
I am sorry if this has been posted already. I tried my best to search for this but could not find this
Thanks
AnanthOlap cache settings
RSRT u2013> choose a query u2013>properties u2013> Cache Mode: Persistent Cache per Application server, Persistence Mode: Flat File.
RSRT
http://www.sap-press.de/download/dateien/1049/sappress_bw_performance_optimization_guide_080.pdf -
An index can not being used and still afect a query performance?
Hi i have a query with a high cost so i created two indexes, A and B, to improve its performance.
After the creation of the indexes when i reviewed the execution plan of the query the cost had been reduced, but i noticed that the index B is not being used,
and if i try to force the query to use index B with a HINT the cost increases, so i decided to drop the index B.
Once i droped the index B i checked the execution plan again and then i noticed that the cost of the query increased, if i recreate the index B the explain plan
shows a lower cost even though its not being used by the execution plan.
Does anyone know why is this happening?
An index can, not being used by the execution plan and still affect a query performance?user11173393 wrote:
Hi i have a query with a high cost so i created two indexes, A and B, to improve its performance.
After the creation of the indexes when i reviewed the execution plan of the query the cost had been reduced, but i noticed that the index B is not being used,
and if i try to force the query to use index B with a HINT the cost increases, so i decided to drop the index B.
Once i droped the index B i checked the execution plan again and then i noticed that the cost of the query increased, if i recreate the index B the explain plan
shows a lower cost even though its not being used by the execution plan.
Does anyone know why is this happening?
An index can, not being used by the execution plan and still affect a query performance?You said that is what is happening, & I believe you. -
Using REGEXP_LIKE to query with exclusion
Dear all,
how to query a column that doesn't contain value with a particular letter in the middle of it using REGEXP_LIKE? Suppose I have the following:
SQL> select * from test;
TEMP
john
joni
jane
johny
jonny
SQL> select temp from test where regexp_like(temp,'[^h]');
TEMP
john
joni
jane
johny
jonny
SQL> select temp from test where regexp_like(temp,'[a-z]+[^h][a-z]+');
TEMP
john
joni
jane
johny
jonnyThe above code tries to query temp that doesn't contain letter 'h' in the middle (so it querries joni, jane, and jonny) but it cannot be done with
regexp_like(temp,'[^h]');
regexp_like(temp,'[a-z]+[^h][a-z]+');since these 2 patterns satisfy all the available rows. What is the solution?
Best regards,
ValFrank Kulash wrote:
Hi,
Valerie Debonair wrote:
Hi Frank,
could you give another example, this time involves not equal character ^ instead of using NOT?Why? What's wrong with NOT?
There are other (more complicated and/or less efficient) ways to get the same results, but whatever objection you have to NOT might apply to those ways, too. Rather than have people suggest solutions one at a time, and wait for you to say "I can't use that", I suggest you explain why you can't use it, so that the next solution won;t have the same problem.
^ doesn't mean "not"; it means "something other than" or "anything except". This problem doesn't involve whther or not the string contains something other than 'h'; it only matters whether or not the string conatins 'h'.Hi Frank,
the purpose of this question is for me to understand the usage of ^ character with REGEXP_LIKE. While your solution solves the particular case that I asked, it didn't fully serve the purpose of the question even though it works well in the real world. In order words, I was focusing on the semantic of regular expression.
Thanks for the help though, your explanation on ^ character sheds me some light....... -
Using DB Links - Improving SELECT query performance
Hi there,
I am using dblink in the following query:
I would like to improve performance of the query by using hints as per described in the link: http://www.experts-exchange.com/Database/Oracle/9.x/Q_23640348.html. However, i am not sure how can i include this in my select query.
Details are:
Oracle - 9i Database Terminal Release .8
DB Link: TCPROD
Could someone please explain with an example how to use hints to get the query to select data on the remote database and then return the results to the target database?
Many Thanks.
SELECT ec.obid AS prObid,
ec.b2ProgramName AS program,
ec.projectName AS project,
ec.wbsID AS prNo,
ec.wbsName AS title,
ec.revision AS revision,
ec.superseded AS revisionSuperseded,
ec.lifeCycleState AS lifeCycleState,
ec.b2ChangeType AS type,
ec.b2Complexity AS subType,
ec.r1SsiCode AS ssi,
ec.b2disposition as disposition,
ec.wbsOriginator AS requestor,
ec.wbsAdministrator AS administrator,
ec.changepriority as priority,
ec.r1tsc as tsc,
ec.t1comments as tenixComments,
ec.b2securityclass as securityClassification,
ec.t1changesafety as safety,
ec.t1actionofficer as actionOfficer,
ec.t1changereason as changeReason,
ec.t1wbsextchangenumber as extChangeNo,
ec.creator as creator,
to_date(substr(ec.creationdate,
0,
instr(ec.creationdate, ':', 1, 3) - 1),
'YYYY/MM/DD-HH24:MI:SS') as creationdate,
to_date(ec.originatorassigndate, 'YYYY/MM/DD') as originatorassigndate,
zbd.description as description,
zbc.comments as comments
FROM (SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM awdbt1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM mart1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM mpsdt1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM nondt1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnast1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnlht1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnolt1m4.cmPrRpIt@TCPROD
UNION
SELECT obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rzptt1m4.cmPrRpIt@TCPRODit's the tablename in the hint, not the column name
something like
SELECT ec.obid AS prObid,
ec.b2ProgramName AS program,
ec.projectName AS project,
ec.wbsID AS prNo,
ec.wbsName AS title,
ec.revision AS revision,
ec.superseded AS revisionSuperseded,
ec.lifeCycleState AS lifeCycleState,
ec.b2ChangeType AS type,
ec.b2Complexity AS subType,
ec.r1SsiCode AS ssi,
ec.b2disposition as disposition,
ec.wbsOriginator AS requestor,
ec.wbsAdministrator AS administrator,
ec.changepriority as priority,
ec.r1tsc as tsc,
ec.t1comments as tenixComments,
ec.b2securityclass as securityClassification,
ec.t1changesafety as safety,
ec.t1actionofficer as actionOfficer,
ec.t1changereason as changeReason,
ec.t1wbsextchangenumber as extChangeNo,
ec.creator as creator,
to_date(substr(ec.creationdate,
0,
instr(ec.creationdate, ':', 1, 3) - 1),
'YYYY/MM/DD-HH24:MI:SS') as creationdate,
to_date(ec.originatorassigndate, 'YYYY/MM/DD') as originatorassigndate
FROM (SELECT /*+ DRIVING_SITE(awdbt1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM awdbt1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(mart1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM mart1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(mpsdt1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM mpsdt1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(nondt1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM nondt1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(rnast1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnast1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(rnlht1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnlht1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(rnolt1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rnolt1m4.cmPrRpIt@TCPROD
UNION
SELECT /*+ DRIVING_SITE(rzptt1m4.cmPrRpIt) */ obid,
b2ProgramName,
projectName,
wbsID,
wbsName,
revision,
superseded,
lifeCycleState,
b2ChangeType,
b2Complexity,
r1SsiCode,
b2disposition,
wbsOriginator,
wbsAdministrator,
changepriority,
r1tsc,
t1comments,
b2securityclass,
t1changesafety,
t1actionofficer,
t1changereason,
t1wbsextchangenumber,
creator,
creationdate,
originatorassigndate
FROM rzptt1m4.cmPrRpIt@TCPROD) ec(not tested, of course) -
Can I refactor this query to use an index more efficiently?
I have a members table with fields such as id, last name, first name, address, join date, etc.
I have a unique index defined on (last_name, join_date, id).
This query will use the index for a range scan, no sort required since the index will be in order for that range ('Smith'):
SELECT members.*
FROM members
WHERE last_name = 'Smith'
ORDER BY joindate, idIs there any way I can get something like the following to use the index (with no sort) as well:
SELECT members.*
FROM members
WHERE last_name like 'S%'
ORDER BY joindate, idI understand the difficulty is probably; even if it does a range scan on every last name 'S%' (assuming it can?), they're not necessarily in order. Case in point:
Last_Name: JoinDate:
Smith 2/5/2010
Smuckers 1/10/2010An index range scan of 'S%' would return them in the above order, which is not ordered by joindate.
So is there any way I can refactor this (query or index) such that the index can be range scanned (using LIKE 'x%') and return rows in the correct order without performing a sort? Or is that simply not possible?xaeryan wrote:
I have a members table with fields such as id, last name, first name, address, join date, etc.
I have a unique index defined on (last_name, join_date, id).
This query will use the index for a range scan, no sort required since the index will be in order for that range ('Smith'):
SELECT members.*
FROM members
WHERE last_name = 'Smith'
ORDER BY joindate, idIs there any way I can get something like the following to use the index (with no sort) as well:
SELECT members.*
FROM members
WHERE last_name like 'S%'
ORDER BY joindate, idI understand the difficulty is probably; even if it does a range scan on every last name 'S%' (assuming it can?), they're not necessarily in order. Case in point:
Last_Name: JoinDate:
Smith 2/5/2010
Smuckers 1/10/2010An index range scan of 'S%' would return them in the above order, which is not ordered by joindate.
So is there any way I can refactor this (query or index) such that the index can be range scanned (using LIKE 'x%') and return rows in the correct order without performing a sort? Or is that simply not possible?Come on. Index column order does matter. "LIKE 'x%'" actually is full table scan. The db engine accesses contiguous index entries and then uses the ROWID values in the index to retrieve the table rows. -
Report burst:To increase query performance in xcelsius
Is there anyway to increase query performance in xcelsius by using report bursting
Fremlin,
Report bursting is only for distributing your reports to your end users.
You can improve performance only by following the [Best practices|https://www.sdn.sap.com/irj/boc/index?rid=/library/uuid/a084a11c-6564-2b10-79ac-cc1eb3f017ac] in xcelsius.
-Anil -
QUERY PERFORMANCE AND DATA LOADING PERFORMANCE ISSUES
WHAT ARE QUERY PERFORMANCE ISSUES WE NEED TO TAKE CARE PLEASE EXPLAIN AND LET ME KNOW T CODES...PLZ URGENT
WHAT ARE DATALOADING PERFORMANCE ISSUES WE NEED TO TAKE CARE PLEASE EXPLAIN AND LET ME KNOW T CODES PLZ URGENT
WILL REWARD FULL POINT S
REGARDS
GURUBW Back end
Some Tips -
1)Identify long-running extraction processes on the source system. Extraction processes are performed by several extraction jobs running on the source system. The run-time of these jobs affects the performance. Use transaction code SM37 Background Processing Job Management to analyze the run-times of these jobs. If the run-time of data collection jobs lasts for several hours, schedule these jobs to run more frequently. This way, less data is written into update tables for each run and extraction performance increases.
2)Identify high run-times for ABAP code, especially for user exits. The quality of any custom ABAP programs used in data extraction affects the extraction performance. Use transaction code SE30 ABAP/4 Run-time Analysis and then run the analysis for the transaction code RSA3 Extractor Checker. The system then records the activities of the extraction program so you can review them to identify time-consuming activities. Eliminate those long-running activities or substitute them with alternative program logic.
3)Identify expensive SQL statements. If database run-time is high for extraction jobs, use transaction code ST05 Performance Trace. On this screen, select ALEREMOTE user and then select SQL trace to record the SQL statements. Identify the time-consuming sections from the results. If the data-selection times are high on a particular SQL statement, index the DataSource tables to increase the performance of selection (see no. 6 below). While using ST05, make sure that no other extraction job is running with ALEREMOTE user.
4)Balance loads by distributing processes onto different servers if possible. If your site uses more than one BW application server, distribute the extraction processes to different servers using transaction code SM59 Maintain RFC Destination. Load balancing is possible only if the extraction program allows the option
5)Set optimum parameters for data-packet size. Packet size affects the number of data requests to the database. Set the data-packet size to optimum values for an efficient data-extraction mechanism. To find the optimum value, start with a packet size in the range of 50,000 to 100,000 and gradually increase it. At some point, you will reach the threshold at which increasing packet size further does not provide any performance increase. To set the packet size, use transaction code SBIW BW IMG Menu on the source system. To set the data load parameters for flat-file uploads, use transaction code RSCUSTV6 in BW.
6)Build indexes on DataSource tables based on selection criteria. Indexing DataSource tables improves the extraction performance, because it reduces the read times of those tables.
7)Execute collection jobs in parallel. Like the Business Content extractors, generic extractors have a number of collection jobs to retrieve relevant data from DataSource tables. Scheduling these collection jobs to run in parallel reduces the total extraction time, and they can be scheduled via transaction code SM37 in the source system.
8). Break up your data selections for InfoPackages and schedule the portions to run in parallel. This parallel upload mechanism sends different portions of the data to BW at the same time, and as a result the total upload time is reduced. You can schedule InfoPackages in the Administrator Workbench.
You can upload data from a data target (InfoCube and ODS) to another data target within the BW system. While uploading, you can schedule more than one InfoPackage with different selection options in each one. For example, fiscal year or fiscal year period can be used as selection options. Avoid using parallel uploads for high volumes of data if hardware resources are constrained. Each InfoPacket uses one background process (if scheduled to run in the background) or dialog process (if scheduled to run online) of the application server, and too many processes could overwhelm a slow server.
9). Building secondary indexes on the tables for the selection fields optimizes these tables for reading, reducing extraction time. If your selection fields are not key fields on the table, primary indexes are not much of a help when accessing data. In this case it is better to create secondary indexes with selection fields on the associated table using ABAP Dictionary to improve better selection performance.
10)Analyze upload times to the PSA and identify long-running uploads. When you extract the data using PSA method, data is written into PSA tables in the BW system. If your data is on the order of tens of millions, consider partitioning these PSA tables for better performance, but pay attention to the partition sizes. Partitioning PSA tables improves data-load performance because it's faster to insert data into smaller database tables. Partitioning also provides increased performance for maintenance of PSA tables for example, you can delete a portion of data faster. You can set the size of each partition in the PSA parameters screen, in transaction code SPRO or RSCUSTV6, so that BW creates a new partition automatically when a threshold value is reached.
11)Debug any routines in the transfer and update rules and eliminate single selects from the routines. Using single selects in custom ABAP routines for selecting data from database tables reduces performance considerably. It is better to use buffers and array operations. When you use buffers or array operations, the system reads data from the database tables and stores it in the memory for manipulation, improving performance. If you do not use buffers or array operations, the whole reading process is performed on the database with many table accesses, and performance deteriorates. Also, extensive use of library transformations in the ABAP code reduces performance; since these transformations are not compiled in advance, they are carried out during run-time.
12)Before uploading a high volume of transaction data into InfoCubes, activate the number-range buffer for dimension IDs. The number-range buffer is a parameter that identifies the number of sequential dimension IDs stored in the memory. If you increase the number range before high-volume data upload, you reduce the number of reads from the dimension tables and hence increase the upload performance. Do not forget to set the number-range values back to their original values after the upload. Use transaction code SNRO to maintain the number range buffer values for InfoCubes.
13)Drop the indexes before uploading high-volume data into InfoCubes. Regenerate them after the upload. Indexes on InfoCubes are optimized for reading data from the InfoCubes. If the indexes exist during the upload, BW reads the indexes and tries to insert the records according to the indexes, resulting in poor upload performance. You can automate the dropping and regeneration of the indexes through InfoPackage scheduling. You can drop indexes in the Manage InfoCube screen in the Administrator Workbench.
14)IDoc (intermediate document) archiving improves the extraction and loading performance and can be applied on both BW and R/3 systems. In addition to IDoc archiving, data archiving is available for InfoCubes and ODS objects.
Hope it Helps
Chetan
@CP.. -
Query Performance issue in Oracle Forms
Hi All,
I am using oracle 9i DB and forms 6i.
In query form ,qry took long time to load the data into form.
There are two tables used here.
1 table(A) contains 5 crore records another table(B) has 2 crore records.
The recods fetching range 1-500 records.
Table (A) has no index on main columns,after created the index on main columns in table A ,the query is fetched the data quickly.
But DBA team dont want to create index on table A.Because of table space problem.
If create the index on main table (A) ,then performance overhead in production.
Concurrent user capacity is 1500.
Is there any alternative methods to handle this problem.
Regards,
RS1) What is a crore? Wikipedia seems to indicate that it's either 10,000,000 or 500,000
http://en.wikipedia.org/wiki/Crore
I'll assume that we're talking about tables with 50 million and 20 million rows, respectively.
2) Large tables with no indexes are definitely going to be slow. If you don't have the disk space to create an appropriate index, surely the right answer is to throw a bit of disk into the system.
3) I don't understand the comment "If create the index on main table (A) ,then performance overhead in production." That seems to contradict the comment you made earlier that the query performs well when you add the index. Are you talking about some other performance overhead?
Justin -
How to improve query performance built on a ODS
Hi,
I've built a report on FI_GL ODS (BW3.5). The report execution time takes almost 1hr.
Is there any method to improve or optimize th query performance that build on ODS.
The ODS got huge volume of data ~ 300 Million records for 2 years.
Thanx in advance,
Guru.Hi Raj,
Here are some few tips which helps you in improving ur query performance
Checklist for Query Performance
1. If exclusions exist, make sure they exist in the global filter area. Try to remove exclusions by subtracting out inclusions.
2. Use Constant Selection to ignore filters in order to move more filters to the global filter area. (Use ABAPer to test and validate that this ensures better code)
3. Within structures, make sure the filter order exists with the highest level filter first.
4. Check code for all exit variables used in a report.
5. Move Time restrictions to a global filter whenever possible.
6. Within structures, use user exit variables to calculate things like QTD, YTD. This should generate better code than using overlapping restrictions to achieve the same thing. (Use ABAPer to test and validate that this ensures better code).
7. When queries are written on multiproviders, restrict to InfoProvider in global filter whenever possible. MultiProvider (MultiCube) queries require additional database table joins to read data compared to those queries against standard InfoCubes (InfoProviders), and you should therefore hardcode the infoprovider in the global filter whenever possible to eliminate this problem.
8. Move all global calculated and restricted key figures to local as to analyze any filters that can be removed and moved to the global definition in a query. Then you can change the calculated key figure and go back to utilizing the global calculated key figure if desired
9. If Alternative UOM solution is used, turn off query cache.
10. Set read mode of query based on static or dynamic. Reading data during navigation minimizes the impact on the R/3 database and application server resources because only data that the user requires will be retrieved. For queries involving large hierarchies with many nodes, it would be wise to select Read data during navigation and when expanding the hierarchy option to avoid reading data for the hierarchy nodes that are not expanded. Reserve the Read all data mode for special queriesu2014for instance, when a majority of the users need a given query to slice and dice against all dimensions, or when the data is needed for data mining. This mode places heavy demand on database and memory resources and might impact other SAP BW processes and tasks.
11. Turn off formatting and results rows to minimize Frontend time whenever possible.
12. Check for nested hierarchies. Always a bad idea.
13. If "Display as hierarchy" is being used, look for other options to remove it to increase performance.
14. Use Constant Selection instead of SUMCT and SUMGT within formulas.
15. Do review of order of restrictions in formulas. Do as many restrictions as you can before
calculations. Try to avoid calculations before restrictions.
17. Turn off warning messages on queries.
18. Check to see if performance improves by removing text display (Use ABAPer to test and validate that this ensures better code).
19. Check to see where currency conversions are happening if they are used.
20. Check aggregation and exception aggregation on calculated key figures. Before aggregation is generally slower and should not be used unless explicitly needed.
21. Avoid Cell Editor use if at all possible.
22. Make sure queries are regenerated in production using RSRT after changes to statistics, consistency changes, or aggregates.
23. Within the free characteristics, filter on the least granular objects first and make sure those come first in the order.
Maybe you are looking for
-
IMac goes blank screen and freezes
Hi, I got 24" iMac running OS X 10.5 and Win 7. Last night I was running windows playing some game when screen gone totaly black for unknown reason and iMac froze. After that booting to OSX it caused grey screen just after a couple of seconds when th
-
Why is the search so poor? The first page had nothing useful when I typed: owner change in finder Google would do much better. Xserve Other OS
-
How do I update Lightroom 4.4 to download canon g16 raw images?
How do I update Lightroom 4.4 to download canon g16 raw images from my camera?
-
Keyboard Shortcuts work in Preview, but not in .SWF (Captivate 5)
I'm working on a project where keyboard shortcuts are the *main* way users will advance the screen. This is because keyboard shortcuts are the main way to accomplish tasks in the software I am teaching. Most of my keyboard shortcuts use an invisble
-
Table to see the GRC request description
Hello Experts, Do you know the table in which GRC request description stores? I checked GRAC* tables but couldn't find one. Thanks in advance Hari