CCM performance problems table info needed!!
Hello,
We have iimpemented SRM50 wit hCCM 200_700 level 6. We have serious performance problems when uploading content and publishing the catalogs. Does anybody know, as we have 29 procurement catalogs, if for all catalogs the entres are entered in the table /CCM/D_ITM.
We have 29 catalogs but only 4000 items so we think the entries are double. How many items and catalogs should CCM be able to handle ?
Thanks of course i will reward points
Kind regards,
Antoinette Stork
Hi,
See this related thread/notes:
Re: items in catalog not flagged as "approved"
Note 919725 - BBP_CCM_TRANSFER_CATALOG - Performance Improvement
BR,
Disha.
Do reward points for useful answers.
Similar Messages
-
Performance problems, do we need to upgrade. Server and Database level
Problem:
I'm a Java programmer and a Transact SQL DBA. So i have knowledge about databases. Nowwe have a database who performs very bad and got much deadlock problems and so on. It's an Oracle Database.
We have Oracle version 9 and an application in Delphi. The bad performance is only since a while. We have cleaned the archive.
My suggestion is why not migrate to a newer version of Oracle. Change some hardware specs get up to date.Then i think we will have less problems.
But ofcourse this is more trail on error. That why i hope there is an Oracle specialist here who can help me with a few questions.
Users and Specs
I got 150 till 180 users
i got a server with 1 processor XEON 233 GHZ
4 gig memory, constant use 1,5 gig
Questions
1. Is it a good idea to upgrade? Maybe not to solve al the problems, but version 9 is old, there is version 10 or 11.
2. Which version we should use 10 or 11? 11 is in use for a while so this sounds like a good idea.
3. Are the specs OK or must i do something about the server to?
Maybe dual core, or Enterprise (64 bit). Memory upgrade?
4. Maybe for 64 bit i need Oracle version 11 to have good support on it?
I hope somebody can help me a bit.
Thanks,
Kind regards,
AndréHi Andre,
. Is it a good idea to upgrade? Maybe not to solve al the problems, but version 9 is old, there is version 10 or 11.
2. Which version we should use 10 or 11? 11 is in use for a while so this sounds like a good idea.I suggest you to upgrade to latest available 11.2.0.2.
But do complete testing your upgraded database before you move to PRODUCTION.
. Are the specs OK or must i do something about the server to?It all depends on the usage and concurrent users :)
4. Maybe for 64 bit i need Oracle version 11 to have good support on it?Regardless of bit version all Oracle Versions has good support.
Refer MOS tech notes:
*How to Perform a Full Database Export Import during Upgrade, Migrate, Copy, or Move of a Database [ID 286775.1]*
*Minimizing Downtime During Production Upgrade [ID 478308.1]*
*Different Upgrade Methods For Upgrading Your Database [ID 419550.1]*
thanks,
X A H E E R -
Hi!
We use SAP Solution Manager 4.0, SPS 13 for System Monitoring of almost 100 systems.
We configured SOLMAN as CEN system and activated the CCMSPING with push option.
For almost the 2/3 of the systems we configured SAPCCM4X agents.
Our current problem is a performance issue in tcode DSWP. When we try to enter into one of the solutions there (by pressing on it) we get the following warning:
Determining warning messages from CCMS
It tooks a while and then it is possible to enter into the solution.
My important question is of course:
How can this issue be solved?
Is there some techniques to analyse and change the problem
As I know there are two ways of sending/retrieving the data:
via RFC (pull)
via CCMS (push)
Which method is faster?
Is there some changes in CCMSPING possible? (interval for sending/retrieving the data)?
Is there some other settings (like profile parameters e.g. rdisp/...) that could be performed to solve the problem?
Thank you very much for any helpful information
Regards
ThomHi Markus!
>Determining warning messages from CCMS
>It tooks a while and then it is possible to enter into the solution.
The SOLMAN asks for monitoring data alls the solutions to get this data in tcode DSWP.
>The interesting question is: What is the system doing? What do you see in SM50/SM66 when you try to >get into a solution and the system displays that message?
The two involving dialog processes are:
1) DIAG, status: on hold, Report: SAPLSALC, user: <my_user>
2) DIAG, staus: running, Report: SAPMSSY0, user: CSMREG
Do you have enough information to give some suggestions?
Thank you
regards
Thom -
Performance Problems with "For all Entries" and a big internal table
We have big Performance Problems with following Statement:
SELECT * FROM zeedmt_zmon INTO TABLE gt_zmon_help
FOR ALL ENTRIES IN gt_zmon_help
WHERE
status = 'IAI200' AND
logdat IN gs_dat AND
ztrack = gt_zmon_help-ztrack.
In the internal table gt_zmon_help are over 1000000 entries.
Anyone an Idea how to improve the Performance?
Thank you!>
Matthias Weisensel wrote:
> We have big Performance Problems with following Statement:
>
>
SELECT * FROM zeedmt_zmon INTO TABLE gt_zmon_help
> FOR ALL ENTRIES IN gt_zmon_help
> WHERE
> status = 'IAI200' AND
> logdat IN gs_dat AND
> ztrack = gt_zmon_help-ztrack.
>
> In the internal table gt_zmon_help are over 1000000 entries.
> Anyone an Idea how to improve the Performance?
>
> Thank you!
You can't expect miracles. With over a million entries in your itab any select is going to take a bit of time. Do you really need all these records in the itab? How many records is the select bringing back? I'm assuming that you have got and are using indexes on your ZEEDMT_ZMON table.
In this situation, I'd first of all try to think of another way of running the query and restricting the amount of data, but if this were not possible I'd just run it in the background and accept that it is going to take a long time. -
Performance problem with table COSS...
Hi
Anyone encountered performance problem with these table : COSS, COSB, COEP
Best Regards>
gsana sana wrote:
> Hi Guru's
>
> this is the select Query which is taking much time in Production. so please help me to improve the preformance with BSEG.
>
> this is my select query:
>
> select bukrs
> belnr
> gjahr
> bschl
> koart
> umskz
> shkzg
> dmbtr
> ktosl
> zuonr
> sgtxt
> kunnr
> from bseg
> into table gt_bseg1
> for all entries in gt_bkpf
> where bukrs eq p_bukrs
> and belnr eq gt_bkpf-belnr
> and gjahr eq p_gjahr
> and buzei in gr_buzei
> and bschl eq '40'
> and ktosl ne 'BSP'.
>
> UR's
> GSANA
Hi,
This is what I know and please if any expert think its incorrect, please do correct me.
BSEG is a cluster table with BUKRS, BELNR, GJAHR and BUZEI as the key whereas other key will be stored in database as raw data thus SAP apps will need to convert that raw data first if we are using other keys in where condition. Hence, I suggest to use up to buzei in the where condition and filter other condition in internal table level like using Delete statement. Hope its help.
Regards,
Abraham -
Performance problem: 1.000 queries over a 1.000.000 rows table
Hi everybody!
I have a difficult performance problem: I use JDBC over an ORACLE database. I need to build a map using data from a table with around 1.000.000 rows. My query is very simple (see the code) and takes an average of 900 milliseconds, but I must perform around 1.000 queries with different parameters. The final result is that user must wait several minutes (plus the time needed to draw the map and send it to the client)
The code, very simplified, is the following:
String sSQLCreateView =
"CREATE VIEW " + sViewName + " AS " +
"SELECT RIGHT_ASCENSION, DECLINATION " +
"FROM T_EXO_TARGETS " +
"WHERE (RIGHT_ASCENSION BETWEEN " + dRaMin + " AND " + dRaMax + ") " +
"AND (DECLINATION BETWEEN " + dDecMin + " AND " + dDecMax + ")";
String sSQLSentence =
"SELECT COUNT(*) FROM " + sViewName +
" WHERE (RIGHT_ASCENSION BETWEEN ? AND ?) " +
"AND (DECLINATION BETWEEN ? AND ?)";
PreparedStatement pstmt = in_oDbConnection.prepareStatement(sSQLSentence);
for (int i = 0; i < 1000; i++)
pstmt.setDouble(1, a);
pstmt.setDouble(2, b);
pstmt.setDouble(3, c);
pstmt.setDouble(4, d);
ResultSet rset = pstmt.executeQuery();
X = rset.getInt(1);
I have yet created index with RIGHT_ASCENSION and DECLINATION fields (trying different combinations).
I have tried yet multi-threads, with very bad results
Has anybody a suggestion ?
Thank you very much!How many total rows are there likely to be in the View you create?
Perhaps just do a select instead of a view, and loop thru the resultset totalling the ranges in java instead of trying to have 1000 queries do the job. Something like:
int iMaxRanges = 1000;
int iCount[] = new int[iMaxRanges];
class Range implements Comparable
float fAMIN;
float fAMAX;
float fDMIN;
float fDMAX;
float fDelta;
public Range(float fASC_MIN, float fASC_MAX, float fDEC_MIN, float fDEC_MAX)
fAMIN = fASC_MIN;
fAMAX = fASC_MAX;
fDMIN = fDEC_MIN;
fDMAX = fDEC_MAX;
public int compareTo(Object range)
Range comp = (Range)range;
if (fAMIN < comp.fAMIN)
return -1;
if (fAMAX > comp.fAMAX)
return 1;
if (fDMIN < comp.fDMIN)
return -1;
if (fDMAX > comp.fDMAX)
return 1;
return 0;
List listRanges = new ArrayList(iMaxRanges);
listRanges.add(new Range(1.05, 1.10, 120.5, 121.5));
//...etc.
String sSQL =
"SELECT RIGHT_ASCENSION, DECLINATION FROM T_EXO_TARGETS " +
"WHERE (RIGHT_ASCENSION BETWEEN " + dRaMin + " AND " + dRaMax + ") " +
"AND (DECLINATION BETWEEN " + dDecMin + " AND " + dDecMax + ")";
Statement stmt = in_oDbConnection.createStatement();
ResultSet rset = stmt.executeQuery(sSQL);
while (rset.next())
float fASC = rset.getFloat("RIGHT_ASCENSION");
flaot fDEC = rset.getFloat("DECLINATION");
int iRange = Collections.binarySearch(listRanges, new Range(fASC, fASC, fDEC, fDEC));
if (iRange >= 0)
++iCount[iRange]; -
Performance problem when printing - table TSPEVJOB big size
Hi,
in a SAP ERP system there is performance problem when printing because table TSPEVJOB has many millions of entries.
This table has reached a considerable size (about 90 Gb); db size is 200 Gb.
Standard reports have been scheduled correctly and kernel of SAP ERP system is updated.
I've tried to scheduled report RSPO1041 (instead of RSPO0041) to reduce entries but it is not possible run it during normal operations because it locks the printing system.
Do you know why this table has increased ?
Are there any reports to clean this table or other methods ?
Thanks.
Maurizio ManeraDear,
Please see the Note 706478 - Preventing Basis tables from increasing considerably and
Note 195157 - Application log: Deletion of logs.
Note 1566659 - Table TSPEVJOB is filled excessively
Note 1293472 - Spool work process hangs on semaphore 43
Note 1362343 - Deadlocks in table TSP02 and TSPEVJOB
Note 996455 - Deadlocks on TSP02 or TSPEVJOB when you delete
For more information see the below link as,
http://www.sqlservercurry.com/2008/04/how-to-delete-records-from-large-table.html
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=91179
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=83525
If any doubts Let me know.
Thomas
Edited by: thomas_raja on Aug 7, 2011 12:29 PM -
Performance problem(ANEA/ANEP table) in Select statement
Hi
I am using below select statement to fetch data.
Does the below where statement have performance issue?
can you Pls suggest.
1)In select of ANEP table, i am not using all the Key field in where condition. will it have performance problem?
2)does the order of where condition should be same as in table, if any one field order change also will have effect performance
SELECT bukrs
anln1
anln2
afabe
gjahr
peraf
lnran
bzdat
bwasl
belnr
buzei
anbtr
lnsan
FROM anep
INTO TABLE o_anep
FOR ALL ENTRIES IN i_anla
WHERE bukrs = i_anla-bukrs
AND anln1 = i_anla-anln1
AND anln2 = i_anla-anln2
AND afabe IN s_afabe
AND bzdat =< p_date
AND bwasl IN s_bwasl.
SELECT bukrs
anln1
anln2
gjahr
lnran
afabe
aufwv
nafal
safal
aafal
erlbt
aufwl
nafav
aafav
invzv
invzl
FROM anea
INTO TABLE o_anea
FOR ALL ENTRIES IN o_anep
WHERE bukrs = o_anep-bukrs
AND anln1 = o_anep-anln1
AND anln2 = o_anep-anln2
AND gjahr = o_anep-gjahr
AND lnran = o_anep-lnran
AND afabe = o_anep-afabe.
Moderator message: Please Read before Posting in the Performance and Tuning Forum
Edited by: Thomas Zloch on Aug 9, 2011 9:37 AM1. Yes. If you have only a few primary keys in youe WHERE condition that does affect the performance. But some times requirement itself may be in that way. We may not be knowing all the primary keys to given them in WHER conditon. If you know the values, then provide them without fail.
2. Yes. It's better to always follow the sequence in WHERE condition and even in the fields being fetched.
One important point is, whenever you use FOR ALL ENTRIES IN, please make sure that the itab IS NOT INITIAL i.e. the itab must have been filled in. So, place the same conditin before both the SELECT queries like:
IF i_anla[] IS NOT INITIAL.
SELECT bukrs
anln1
anln2
afabe
gjahr
peraf
lnran
bzdat
bwasl
belnr
buzei
anbtr
lnsan
FROM anep
INTO TABLE o_anep
FOR ALL ENTRIES IN i_anla
WHERE bukrs = i_anla-bukrs
AND anln1 = i_anla-anln1
AND anln2 = i_anla-anln2
AND afabe IN s_afabe
AND bzdat =< p_date
AND bwasl IN s_bwasl.
ENDIF.
IF o_anep[] IS NOT INITIAL.
SELECT bukrs
anln1
anln2
gjahr
lnran
afabe
aufwv
nafal
safal
aafal
erlbt
aufwl
nafav
aafav
invzv
invzl
FROM anea
INTO TABLE o_anea
FOR ALL ENTRIES IN o_anep
WHERE bukrs = o_anep-bukrs
AND anln1 = o_anep-anln1
AND anln2 = o_anep-anln2
AND gjahr = o_anep-gjahr
AND lnran = o_anep-lnran
AND afabe = o_anep-afabe.
ENDIF. -
Performance problem with query on bkpf table
hi good morning all ,
i ahave a performance problem with a below query on bkpf table .
SELECT bukrs
belnr
gjahr
FROM bkpf
INTO TABLE ist_bkpf_temp
WHERE budat IN s_budat.
is ther any possibility to improve the performanece by using index .
plz help me ,
thanks in advance ,
regards ,
srinivashi,
if u can add bukrs as input field or if u have bukrs as part of any other internal table to filter out the data u can use:
for ex:
SELECT bukrs
belnr
gjahr
FROM bkpf
INTO TABLE ist_bkpf_temp
WHERE budat IN s_budat
and bukrs in s_bukrs.
or
SELECT bukrs
belnr
gjahr
FROM bkpf
INTO TABLE ist_bkpf_temp
for all entries in itab
WHERE budat IN s_budat
and bukrs = itab-bukrs.
Just see , if it is possible to do any one of the above?? It has to be verified with ur requirement. -
Performance problems - query on the copy of a table is faster than the orig
Hi all,
I have sql (select) performance problems with a specific table (costs 7800)(Oracle 10.2.0.4.0, Linux).
So I copied the table 1:1 (same structure, same data, same indexes etc. ) under the same area (database, user, tablespace etc.) and gathered the table_stats with the same parameters. The same query on this copied table is faster and the costs are just 3600.
Why for gods sake is the query on this new table faster??
I appreciate any idea.
Thank you!
Fibo
Edited by: user954903 on 13.01.2010 04:23Could you please share more information/link which can elaborate the significance of using SHRINK clause.
If this is so useful and can shrink the unused space , why not my database architect has suggested this :).
Any disadvantage also?It moves the highwater mark back to the lowest position, therefore full tables scans would work faster in some cases. Also it can reduce number of migrated rows and number of used blocks.
Disadvantage is that it involves row movement, so operations which based on rowid are permitted during the shrinking.
I think it is even better to stop all operations on the table when shrinking and disable all triggers. Another problem is that this process can take a long time.
Guru's, please correct me if I'm mistaken.
Edited by: Oleg Gorskin on 13.01.2010 5:50 -
I have upgraded my MacBook Pro 15 late 2011 to 8Gb RAM, now when I go to about this mac in it more info it doesn't show AMD graphics card as it used to before, it just show Intel HD Graphics 3000 512 MB is that a problem that I need to worry about?
and in system report >> hardware >> Graphics/Display >> it shows both grafics card listed.Some Mac's when the RAM is increased the Intel HD 3000 integrated graphics will bump itself up to the higher video RAM it takes, this seems to be what occured.
It also means the dedicated graphics card will be used less, but only so slightly less.
You obviously still should have both, you can test this by turning off Graphics Switching in Energy Saver, that will use the decicated AMD card all the time.
You can also run Cinebench and compare the scores here
Mac video card performance -
Hard disc indicator light on right front of laptop lights only periodically at startup or restart. There appears not to be any performance problems. Do I have a problem in the making? Danger in the future? Need service? Or did I just choose the wrong option somewhere?
The white LED is not a disk activity or power indicator. MacBooks don't have either of these. The indicator will only light as mentioned by Dave Stowe... when your display is asleep it will come on and when your system is asleep, it will pulsate. From what you have indicated, it seems to be working correctly.
-
I am having performance problems executing a query.
System:
Windows 2003 EE
Oracle 9i version 9.2.0.6
DETAIL table with 120Million rows partitioned in 19 partitions by SD_DATEKEY field
We are trying to retrieve the info from an account (SD_KEY) ordered by date (SD_DATEKEY). This account has about 7000 rows and it takes about 1 minute to return the first 100 rows ordered by SD_DATEKEY. This time should be around 5 seconds to be acceptable.
There is a partitioned index by SD_KEY and SD_DATEKEY.
This is the query:
SELECT * FROM DETAIL WHERE SD_KEY = 'xxxxxxxx' AND ROWNUM < 101 ORDER BY SD_DATEKEY
The problem is that all the 7000 rows are read prior to be ordered. I think that it is not necessary for the optimizer to access all the partitions to read all the rows because only the first 100 are needed and the partitions are bounded by SD_DATEKEY.
Any idea to accelerate this query? I know that including a WHERE clause for SD_DATEKEY will increase the performance but I need the first 100 rows and I don't know the date to limit the query.
Anybody knows if this time is a normal response time for tis query or should it be improved?
Thank to all in advance for the future help.Thank to all for the replies.
- We have computed statistics and no changes in the response time.
- We are discussing about restrict the query to some partitions but for the moment this is not the best solution because we don't know where are the latest 100 rows.
- The query from Maurice had the same response time (more or less)
select * from
(SELECT * FROM DETAIL WHERE SD_KEY = 'xxxxxxxx' ORDER BY SD_DATEKEY)
where ROWNUM < 101
- We have a local index on SD_DATEKEY. Do we need another one on SD_KEY? Should it be created as BITMAP?
I can't test inmediately your sugestions because this is a problem with one of our customers. In our test system (that has only 10Million records) the indexes accelerate the query but this is not the same in the customer system. I think the problem is the total records on the table. -
URGENT------MB5B : PERFORMANCE PROBLEM
Hi,
We are getting the time out error while running the transaction MB5B. We have posted the same to SAP global support for further analysis, and SAP revrted with note 1005901 to review.
The note consists of creating the Z table and some Z programs to execute the MB5B without time out error, and SAP has not provided what type logic has to be written and how we can be addressed this.
Could any one suggest us how can we proceed further.
Note as been attached for reference.
Note 1005901 - MB5B: Performance problems
Note Language: English Version: 3 Validity: Valid from 05.12.2006
Summary
Symptom
o The user starts transaction MB5B, or the respective report
RM07MLBD, for a very large number of materials or for all materials
in a plant.
o The transaction terminates with the ABAP runtime error
DBIF_RSQL_INVALID_RSQL.
o The transaction runtime is very long and it terminates with the
ABAP runtime error TIME_OUT.
o During the runtime of transaction MB5B, goods movements are posted
in parallel:
- The results of transaction MB5B are incorrect.
- Each run of transaction MB5B returns different results for the
same combination of "material + plant".
More Terms
MB5B, RM07MLBD, runtime, performance, short dump
Cause and Prerequisites
The DBIF_RSQL_INVALID_RSQL runtime error may occur if you enter too many
individual material numbers in the selection screen for the database
selection.
The runtime is long because of the way report RM07MLBD works. It reads the
stocks and values from the material masters first, then the MM documents
and, in "Valuated Stock" mode, it then reads the respective FI documents.
If there are many MM and FI documents in the system, the runtimes can be
very long.
If goods movements are posted during the runtime of transaction MB5B for
materials that should also be processed by transaction MB5B, transaction
MB5B may return incorrect results.
Example: Transaction MB5B should process 100 materials with 10,000 MM
documents each. The system takes approximately 1 second to read the
material master data and it takes approximately 1 hour to read the MM and
FI documents. A goods movement for a material to be processed is posted
approximately 10 minutes after you start transaction MB5B. The stock for
this material before this posting has already been determined. The new MM
document is also read, however. The stock read before the posting is used
as the basis for calculating the stocks for the start and end date.
If you execute transaction MB5B during a time when no goods movements are
posted, these incorrect results do not occur.
Solution
The SAP standard release does not include a solution that allows you to
process mass data using transaction MB5B. The requirements for transaction
MB5B are very customer-specific. To allow for these customer-specific
requirements, we provide the following proposed implementation:
Implementation proposal:
o You should call transaction MB5B for only one "material + plant"
combination at a time.
o The list outputs for each of these runs are collected and at the
end of the processing they are prepared for a large list output.
You need three reports and one database table for this function. You can
store the lists in the INDX cluster table.
o Define work database table ZZ_MB5B with the following fields:
- Material number
- Plant
- Valuation area
- Key field for INDX cluster table
o The size category of the table should be based on the number of
entries in material valuation table MBEW.
Report ZZ_MB5B_PREPARE
In the first step, this report deletes all existing entries from the
ZZ_MB5B work table and the INDX cluster table from the last mass data
processing run of transaction MB5B.
o The ZZ_MB5B work table is filled in accordance with the selected
mode of transaction MB5B:
- Stock type mode = Valuated stock
- Include one entry in work table ZZ_MB5B for every "material +
valuation area" combination from table MBEW.
o Other modes:
- Include one entry in work table ZZ_MB5B for every "material +
plant" combination from table MARC
Furthermore, the new entries in work table ZZ_MB5B are assigned a unique
22-character string that later serves as a key term for cluster table INDX.
Report ZZ_MB5B_MONITOR
This report reads the entries sequentially in work table ZZ_MB5B. Depending
on the mode of transaction MB5B, a lock is executed as follows:
o Stock type mode = Valuated stock
For every "material + valuation area" combination, the system
determines all "material + plant" combinations. All determined
"material + plant" combinations are locked.
o Other modes:
- Every "material + plant" combination is locked.
- The entries from the ZZ_MB5B work table can be processed as
follows only if they have been locked successfully.
- Start report RM07MLBD for the current "Material + plant"
combination, or "material + valuation area" combination,
depending on the required mode.
- The list created is stored with the generated key term in the
INDX cluster table.
- The current entry is deleted from the ZZ_MB5B work table.
- Database updates are executed with COMMIT WORK AND WAIT.
- The lock is released.
- The system reads the next entry in the ZZ_MB5B work table.
Application
- The lock ensures that no goods movements can be posted during
the runtime of the RM07MLBD report for the "material + Plant"
combination to be processed.
- You can start several instances of this report at the same
time. This method ensures that all "material + plant"
combinations can be processed at the same time.
- The system takes just a few seconds to process a "material +
Plant" combination so there is just minimum disruption to
production operation.
- This report is started until there are no more entries in the
ZZ_MB5B work table.
- If the report terminates or is interrupted, it can be started
again at any time.
Report ZZ_MB5B_PRINT
You can use this report when all combinations of "material + plant", or
"material + valuation area" from the ZZ_MB5B work table have been
processed. The report reads the saved lists from the INDX cluster table and
adds these individual lists to a complete list output.
Estimated implementation effort
An experienced ABAP programmer requires an estimated three to five days to
create the ZZ_MB5B work table and these three reports. You can find a
similar program as an example in Note 32236: MBMSSQUA.
If you need support during the implementation, contact your SAP consultant.
Header Data
Release Status: Released for Customer
Released on: 05.12.2006 16:14:11
Priority: Recommendations/additional info
Category: Consulting
Main Component MM-IM-GF-REP IM Reporting (no LIS)
The note is not release-dependent.
Thanks in advance.
Edited by: Neliea on Jan 9, 2008 10:38 AM
Edited by: Neliea on Jan 9, 2008 10:39 AMbefore you try any of this try working with database-hints as described in note 921165, 902157, 918992
-
Performance problems with XMLTABLE and XMLQUERY involving relational data
Hello-
Is anyone out there using XMLTABLE or XMLQUERY with more than a toy set of data? I am running into serious performance problems tyring to do basic things such as:
* Combine records in 10 relational tables into a single table of XMLTYPE records using XMLTABLE. This hangs indefinitely for any more than 800 records. Oracle has confirmed that this is a problem and is working on a fix.
* Combine a single XMLTYPE record with several relational code tables into a single XMLTYPE record using XMLQUERY and ora:view() to insert code descriptions after each code. Performance is 10 seconds for 10 records (terrible) passing a batch of records , or 160 seconds for one record (unacceptable!). How can it take 10 times longer to process 1/10th the number of records? Ironically, the query plan says it will do a full table scan of records for the batch, but an index access for the one record passed to the XMLQUERY.
I am rapidly losing faith in XML DB, and desparately need some hints on how to work around these performance problems, or at least some assurance that others have been able to get this thing to perform.<Note>Long post, sorry.</Note>
First, thanks for the responses above. I'm impressed with the quality of thought put into them. (Do the forum rules allow me to offer rewards? :) One suggestion in particular made a big performance improvement, and I’m encouraged to hear of good performance in pure XML situations. Unfortunately, I think there is a real performance challenge in two use cases that are pertinent to the XML+relational subject of this post and probably increasingly common as XML DB usage increases:
• Converting legacy tabular data into XML records; and
• Performing code table lookups for coded values in XML records.
There are three things I want to accomplish with this post:
• Clarify what we are trying to accomplish, which might expose completely different approaches than I have tried
• Let you know what I tried so far and the rationale for my approach to help expose flaws in my thinking and share what I have learned
• Highlight remaining performance issues in hopes that we can solve them
What we are trying to accomplish:
• Receive a monthly feed of 10,000 XML records (batched together in text files), each containing information about an employee, including elements that repeat for every year of service. We may need to process an annual feed of 1,000,000 XML records in the future.
• Receive a one-time feed of 500,000 employee records stored in about 10 relational tables, with a maximum join depth of 2 or 3. This is inherently a relational-to-XML process. One record/second is minimally acceptable, but 10 records/sec would be better.
• Consolidate a few records (from different providers) for each employee into a single record. Given the data volume, we need to achieve a minimum rate of 10 records per second. This may be an XML-only process, or XML+relational if code lookups are done during consolidation.
• Allow the records to be viewed and edited, with codes resolved into user-friendly descriptions. Since a user is sitting there, code lookups done when a record is viewed (vs. during consolidation) should not take more than 3 seconds total. We have about 20 code tables averaging a few hundred rows each, though one has 450,000 rows.
As requested earlier, I have included code at the end of this post for example tables and queries that accurately (but simply) replicate our real system.
Why we did and why:
• Stored the source XML records as CLOBS: We did this to preserve the records exactly as they were certified and sent from providers. In addition, we always access the entire XML record as a whole (e.g., when viewing a record or consolidating employee records), so this storage model seemed like a good fit. We can copy them into another format if necessary.
• Stored the consolidated XML employee records as “binary XML”. We did this because we almost always access a single, entire record as a whole (for view/edit), but might want to create some summary statistics at some point. Binary XML seemed the best fit.
• Used ora:view() for both tabular source records and lookup tables. We are not aware of any alternatives at this time. If it made sense, most code tables could be pre-converted into XML documents, but this seemed risky from a performance standpoint because the lookups use both code and date range constraints (the meaning of codes changes over time).
• Stored records as XMLTYPE columns in a table with other key columns on the table, plus an XMLTYPE metadata column. We thought this would facilitate pulling a single record (or a few records for a given employee) quickly. We knew this might be unnecessary given XML indexes and virtual columns, but were not experienced with those and wanted the comfort of traditional keys. We did not used XMLTYPE tables or the XML Repository for documents.
• Used XMLTABLE to consolidate XML records by looping over each distinct employee ID in the source batch. We also tried XMLQUERY and it seems to perform about the same. We can achieve 10 to 20 records/second if we do not do any code lookups during consolidation, just meeting our performance requirement, but still much slower than expected.
• Used PL/SQL with XMLFOREST to convert tabular source records to XML by looping over distinct employee IDs. We tried this outside PL/SQL both with XMLFOREST and XMLTABLE+ora:view(), but it hangs in both cases for more than 800 records (a known/open issue). We were able to get it to work by using an explicit cursor to loop over distinct employee IDs (rather than processing all records at once within the query). The performance is one record/second, which is minimally acceptable and interferes with other database activity.
• Used XMLQUERY plus ora:view() plus XPATH constraints to perform code lookups. When passing a single employee record, the response time ranges from 1 sec to 160 sec depending on the length of the record (i.e., number of years of service). We achieved a 5-fold speedup using an XMLINDEX (thank you Marco!!). The result may be minimally acceptable, but I’m baffled why the index would be needed when processing a single XML record. Other things we tried: joining code tables in the FOR...WHERE clauses, joining code tables using LET with XPATH constraints and LET with WHERE clause constraints, and looking up codes individually via JDBC from the application code at presentation time. All those approaches were slower. Note: the difference I mentioned above in equality/inequality constraint performance was due to data record variations not query plan variations.
What issues remain?
We have a minimally acceptable solution from a performance standpoint with one very awkward PL/SQL workaround. The performance of a mixed XML+relational data query is still marginal IMHO, until we properly utilize available optimizations, fix known problems, and perhaps get some new query optimizations. On the last point, I think the query plan for tabular lookups of codes in XML records is falling short right now. I’m reminded of data warehousing in the days before hash joins and star join optimization. I would be happy to be wrong, and just as happy for viable workarounds if I am right!
Here are the details on our code lookup challenge. Additional suggestions would be greatly appreciated. I’ll try to post more detail on the legacy table conversion challenge later.
-- The main record table:
create table RECORDS (
SSN varchar2(20),
XMLREC sys.xmltype
xmltype column XMLREC store as binary xml;
create index records_ssn on records(ssn);
-- A dozen code tables represented by one like this:
create table CODES (
CODE varchar2(4),
DESCRIPTION varchar2(500)
create index codes_code on codes(code);
-- Some XML records with coded values (the real records are much more complex of course):
-- I think this took about a minute or two
DECLARE
ssn varchar2(20);
xmlrec xmltype;
i integer;
BEGIN
xmlrec := xmltype('<?xml version="1.0"?>
<Root>
<Id>123456789</Id>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
<Element>
<Subelement1><Code>11</Code></Subelement1>
<Subelement2><Code>21</Code></Subelement2>
<Subelement3><Code>31</Code></Subelement3>
</Element>
</Root>
for i IN 1..100000 loop
insert into records(ssn, xmlrec) values (i, xmlrec);
end loop;
commit;
END;
-- Some code data like this (ignoring date ranges on codes):
DECLARE
description varchar2(100);
i integer;
BEGIN
description := 'This is the code description ';
for i IN 1..3000 loop
insert into codes(code, description) values (to_char(i), description);
end loop;
commit;
end;
-- Retrieve one record while performing code lookups. Takes about 5-6 seconds...pretty slow.
-- Each additional lookup (times 3 repeating elements in the data) adds about 1 second.
-- A typical real record has 5 Elements and 20 Subelements, meaning more than 20 seconds to display the record
-- Note we are accessing a single XML record based on SSN
-- Note also we are reusing the one test code table multiple times for convenience of this test
select xmlquery('
for $r in Root
return
<Root>
<Id>123456789</Id>
{for $e in $r/Element
return
<Element>
<Subelement1>
{$e/Subelement1/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement1/Code]/DESCRIPTION/text() }
</Description>
</Subelement1>
<Subelement2>
{$e/Subelement2/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement2/Code]/DESCRIPTION/text()}
</Description>
</Subelement2>
<Subelement3>
{$e/Subelement3/Code}
<Description>
{ora:view("disaac","codes")/ROW[CODE=$e/Subelement3/Code]/DESCRIPTION/text() }
</Description>
</Subelement3>
</Element>
</Root>
' passing xmlrec returning content)
from records
where ssn = '10000';
The plan shows the nested loop access that slows things down.
By contrast, a functionally-similar SQL query on relational data will use a hash join and perform 10x to 100x faster, even for a single record. There seems to be no way for the optimizer to see the regularity in the XML structure and perform a corresponding optimization in joining the code tables. Not sure if registering a schema would help. Using structured storage probably would. But should that be necessary given we’re working with a single record?
Operation Object
|SELECT STATEMENT ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| NESTED LOOPS (SEMI)
| TABLE ACCESS (FULL) CODES
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| XPATH EVALUATION ()
| SORT (AGGREGATE)
| XPATH EVALUATION ()
| TABLE ACCESS (BY INDEX ROWID) RECORDS
| INDEX (RANGE SCAN) RECORDS_SSN
With an xmlindex, the same query above runs in about 1 second, so is about 5x faster (0.2 sec/lookup), which is almost good enough. Is this the answer? Or is there a better way? I’m not sure why the optimizer wants to scan the code tables and index into the (one) XML record, rather than the other way around, but maybe that makes sense if the optimizer wants to use the same general plan as when the WHERE clause constraint is relaxed to multiple records.
-- Add an xmlindex. Takes about 2.5 minutes
create index records_record_xml ON records(xmlrec)
indextype IS xdb.xmlindex;
Operation Object
|SELECT STATEMENT ()
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (GROUP BY)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| TABLE ACCESS (FULL) CODES
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| FILTER ()
| NESTED LOOPS ()
| FAST DUAL ()
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| SORT (AGGREGATE)
| TABLE ACCESS (BY INDEX ROWID) SYS113473_RECORDS_R_PATH_TABLE
| INDEX (RANGE SCAN) SYS113473_RECORDS_R_PATHID_IX
| TABLE ACCESS (BY INDEX ROWID) RECORDS
| INDEX (RANGE SCAN) RECORDS_SSN
Am I on the right path, or am I totally using the wrong approach? I thought about using XSLT but was unsure how to reference the code tables.
I’ve done the best I can constraining the main record to a single row passed to the XMLQUERY. Given Mark’s post (thanks!) should I be joining and constraining the code tables in the SQL WHERE clause too? That’s going to make the query much more complicated, but right now we’re more concerned about performance than complexity.
Maybe you are looking for
-
Just a bit more. According to my laptop, when I hook up my iPhone, either I hit an error of "user defined breakpoint" or it's having issues recognizing the iPad (when the iPhone is hooked up). I've also tried installing iTunes on my work computer, bu
-
With SPD adding 1 month to date not working?
I have a calculated column [1stDayofMth] =DATE(YEAR([Start Date]),MONTH([Start Date]),1) which appears to be working fine. My SPD WF uses the [1stDayofMth] column and does a few calculations to find the next 2 months 1st days [Month 1 Resume Date] =
-
Can't mount as a disk on another Mac (not mine)
I'd like to use my shuffle as a disk and transfer files from my machine to another Mac. This other Mac is an iMac G5 and had iTunes 6.0.2 with OS X.4.4. Everytime I plug in my shuffle, iTunes starts up tells me The iPod "ShuffleName" is linked to ano
-
My iTunes library no longer has many of the songs that were once there. This is not my imagination. For example, I have ten versions of "Stardust" in my iPod, which were of, course, downloaded from my iTunes library. However, I now only have two vers
-
Hi, How to avoid getting empty row when sum() function returns no value if the condition fails in where clause? TIA Harish