SELECT query sometimes runs extremely slowly - UNDO question

Question

Hi,
The Background
We have a subpartitioned table:
CREATE TABLE TAB_A
  RUN_ID           NUMBER                       NOT NULL,
  COB_DATE         DATE                         NOT NULL,
  PARTITION_KEY    NUMBER                       NOT NULL,
  DATA_TYPE        VARCHAR2(10),
  START_DATE       DATE,
  END_DATE         DATE,
  VALUE            NUMBER,
  HOLDING_DATE     DATE,
  VALUE_CURRENCY   VARCHAR2(3),
  NAME             VARCHAR2(60),
PARTITION BY RANGE (COB_DATE)
SUBPARTITION BY LIST (PARTITION_KEY)
SUBPARTITION TEMPLATE
  (SUBPARTITION GROUP1 VALUES (1) TABLESPACE BROIL_LARGE_DATA,
   SUBPARTITION GROUP2 VALUES (2) TABLESPACE BROIL_LARGE_DATA,
   SUBPARTITION GROUP3 VALUES (3) TABLESPACE BROIL_LARGE_DATA,
   SUBPARTITION GROUP4 VALUES (4) TABLESPACE BROIL_LARGE_DATA,
   SUBPARTITION GROUP5 VALUES (DEFAULT) TABLESPACE BROIL_LARGE_DATA
  PARTITION PARTNO_03 VALUES LESS THAN
  (TO_DATE(' 2008-07-22 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
  ( SUBPARTITION PARTNO_03_GROUP1 VALUES (1),
    SUBPARTITION PARTNO_03_GROUP2 VALUES (2),
    SUBPARTITION PARTNO_03_GROUP3 VALUES (3),
    SUBPARTITION PARTNO_03_GROUP4 VALUES (4),
    SUBPARTITION PARTNO_03_GROUP5 VALUES (DEFAULT) ), 
  PARTITION PARTNO_01 VALUES LESS THAN
  (TO_DATE(' 2008-07-23 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
  ( SUBPARTITION PARTNO_01_GROUP1 VALUES (1),
    SUBPARTITION PARTNO_01_GROUP2 VALUES (2),
    SUBPARTITION PARTNO_01_GROUP3 VALUES (3),
    SUBPARTITION PARTNO_01_GROUP4 VALUES (4),
    SUBPARTITION PARTNO_01_GROUP5 VALUES (DEFAULT) ), 
  PARTITION PARTNO_02 VALUES LESS THAN
  (TO_DATE(' 2008-07-24 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIAN'))
  ( SUBPARTITION PARTNO_02_GROUP1 VALUES (1),
    SUBPARTITION PARTNO_02_GROUP2 VALUES (2),
    SUBPARTITION PARTNO_02_GROUP3 VALUES (3),
    SUBPARTITION PARTNO_02_GROUP4 VALUES (4),
    SUBPARTITION PARTNO_02_GROUP5 VALUES (DEFAULT) ), 
  PARTITION PARTNO_OTHER VALUES LESS THAN (MAXVALUE)
  ( SUBPARTITION PARTNO_OTHER_GROUP1 VALUES (1),
    SUBPARTITION PARTNO_OTHER_GROUP2 VALUES (2),
    SUBPARTITION PARTNO_OTHER_GROUP3 VALUES (3),
    SUBPARTITION PARTNO_OTHER_GROUP4 VALUES (4),
    SUBPARTITION PARTNO_OTHER_GROUP5 VALUES (DEFAULT) )
CREATE INDEX TAB_A_IDX ON TAB_A
(RUN_ID, COB_DATE, PARTITION_KEY, DATA_TYPE, VALUE_CURRENCY)
  LOCAL;The table is subpartitioned as each partition typically has 135million rows in it.
Overnight, serveral runs occur that load data into this table (the partitions are rolled over daily, such that the oldest one is dropped and a new one created. Stats are exported from the oldest partition prior to being dropped and imported to the newly created partition. The oldest partition once the partition has been created has it's stats analyzed).
Data loads can load anything from 200 rows to 20million rows into the table, with most of the rows ending up in the Default subpartition. Most of the runs that load a larger set of rows have been set up to add into one of the other 4 subpartitions.
We then run a process to extract data from the table that gets put into a file. This is a two step process (due to Oracle completely picking the wrong execution plan and us not being able to rewrite the query in such a way that it'll pick the right path up by itself!):
1. Identify all the unique currencies
2. Update the (dynamic) sql query to add a CASE clause into the select clause based on the currencies identified in step 1, and run the query.
Step 1 uses this query:
SELECT DISTINCT value_currency
FROM            tab_a
WHERE           run_id = :b3 AND cob_date = :b2 AND partition_key = :b1;and usually finishes this within 20 minutes.
The problem
Occasionally, this simple query runs over 20 minutes (I don't think we've ever seen it run to completion on these occurrences, and I've certainly seen it take over 3 hours before we killed it, for a run where it would normally complete in 2 or 3 minutes), which we've now come to recognise as it "being stuck". The execution path it takes is the same as when it runs normally, there are no unusual wait events, and no unusual wait times. All in all, it looks "normal" except for the fact that it's taking forever (tongue-in-cheek!) to run. When we kill and rerun, the execution time returns to normal. (We've sent system state dumps to Oracle to be analyzed, and they came back with "The database is doing stuff, can't see anything wrong")
We've never been able to come up with any explanation before, but the same run has failed consistently for the last three days, so I managed to wangle a DBA to help me investigate it further.
After looking through the ASH reports, he proposed a theory that the problem was it was having to go to the UNDO to retrieve results, and that this could explain the massive run time of the query.
I looked at the runs and agreed that UNDO might have been used in that particular instance of the query, as another run had loaded data into the table at the same time it was being read.
However, another one of the problematic runs had not had any inserts (or updates/deletes - they don't happen in our process) during the reading of the data, and yet it had taken a long time too. The ASH report showed that it too had read from UNDO.
My question
I understand from this link: http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:44798632736844 about how Selects may generate REDO, but I don't see why UNDO would possibly be generated by a select. Does anyone know of a situation where a select would end up looking through UNDO, even though no inserts/updates/deletes had taken place on the table/index it was looking at?
Also, does the theory that having to look through the UNDO (currently UNDO ts is 50000MB, in case that's relevant) causing queries to take an extremely long time hold water? We're on 10.2.0.3
Message was edited by:
Boneist
Ok, having carried on searching t'internet, I can see that it's maybe Delayed Block Cleanout that's causing the UNDO to be referenced. Even taking that into account, I can't see why going back to the UNDO to be told to commit the change to disk could slow the query down that much? What waits would this show up as, if any?

Answer 1

Since you're on 10.2 and I understand that you use
the statistics of the "previous" content for the
partition that you are now loading (am I right?) you
have to be very careful with the 10g optimizer. If
the statistics tell the optimizer that the values
that you're querying for are sufficiently
out-of-range this might change the execution plan
because it estimates that only a few or no rows will
be returned. So if the old statistics do not fit the
new data loaded in terms of column min/max values
then this could be a valid reason for different
execution plans for some executions (depending on the
statistics of the old partition and the current
values used). Your RUN_ID is a good candidate I guess
as it could be ever increasing... If the max value of
the old partition is sufficiently different from the
current value this might be the cause.
Do you actually use bind variables for that
particular statement? Then we have in addition bind
variable peeking and potentially statement re-using
to consider.
I would prefer literals instead of bind variables or
do you encounter parse issues?Yes, that query runs as part of a procedure and uses bind variables (well, pl/sql variables!). We are aware that because of the histograms that get produced, the stats are not as good as we'd like. I'm wondering if analyzing the partition would be the best way to go, only that means analysing the entire partition, not just the subpartition, I guess? But if other inserts are taking place at the same time, having several analyzes taking place won't help the speed of inserting, or won't it matter?
Do you have the "default" 10g statistics gathering
job active? This could also explain why you get
different execution plans at different execution
times. If the job determines that the statistics of
some of your partitions are stale then it will
attempt to gather statistics even if you already have
statistics imported/generated.No, we turned that off. The stats do not change when we rerun the query - we guess there is some sort of contention taking place, possibly when reading from the UNDO, although I would expect that to show up in the waits - it doesn't appear to though.
Data loads can load anything from 200 rows to
20million rows into the table, with most of therows
ending up in the Default subpartition. Most of the
runs that load a larger set of rows have been setup
to add into one of the other 4 subpartitions.
I'm not sure about above description. Do most of the
rows end up in the default subpartition (most rows in
default partition) or do the larger sets load into
the other 4 ones... (most rows in the non-default
partitions)?Sorry, I mean that the loads that load say 20million + rows at a time have a specified subpartition to go into (defined via some config. We had to make up a "partition key" in order to do this as there is nothing in the data that lends itself to the subpartition list, unfortunately - the process determines which subpartition to load to/extract from via a config table), but this applies to not many of the runs. So, the majority of the runs (with fewer rows) go into the default partition.
The query itself scans the index, not the table, doing partition pruning, etc.
The same SQL_ID doesn't mean it's the same plan. So
are you 100% sure that the plans where the same? I
could imagine (see above) that the execution plans
might be different.The DBA looking at it said that the plans were the same, and I have no reason to doubt him. Also, the session browser in Toad shows the same explain plan in the "Current Statement" tab for normal and abnormal runs and is as follows:
Time     IO Cost     CPU Cost     Cardinality     Bytes     Cost     Plan
                         6     SELECT STATEMENT  ALL_ROWS                    
4 1     4 4     4 82,406     4 1     4 20     4 6          4 HASH UNIQUE                 
3 1     3 4     3 28,686     3 1     3 20     3 5               3 PARTITION RANGE SINGLE  Partition #: 2            
2 1     2 4     2 28,686     2 1     2 20     2 5                    2 PARTITION LIST SINGLE  Partition #: 3       
1 1     1 4     1 28,686     1 1     1 20     1 5                         1 INDEX RANGE SCAN INDEX TAB_A_IDX Access Predicates: "RUN_ID"=:B3 AND "COB_DATE"=:B2 AND "PARTITION_KEY"=:B1  Partition #: 3 
How do you perform your INSERTs? Are overlapping
loads and queries actually working on the same
(sub-)partition of the table or in different ones? Do
you use direct-path inserts or parallel dml?
Direct-path inserts as far I know create "clean"
blocks that do not need a delayed block cleanout.We insert using a select from an external table - there's a parallel hint in there, but I think that is often ignored (at least, I've never seen any hint of sessions running in parallel when looking at the session browser, and I've seen it happen in one of our dev databases, so...). As mentioned above, rows could get inserted into different partitions, although the majority of runs load into the default subpartition. In practise, I don't think more than 3 or 4 loads take place at the same time.
If you loading and querying different partitions then
your queries shouldn't have to check for UNDO except
for the delayed block cleanout case.
You should check at least two important things:
- Are the execution plans different for the slow and
normal executions?
- Get the session statistics (logical I/Os, redo
generated) for the normal and slow ones in order to
see and compare the amount of work that they
generate, and to find out how much redo your query
potentially generated due to delayed block cleanout.It's difficult to do a direct comparison that's exact, due to other work going on in the database, and the abnormal query taking far longer than normal, but here is the ASH comparison between a normal run (1st) and our abnormal run (2nd) (both taken over 90 mins, and the first run may well include other runs that use the same query in the results):
              Exec Time of DB    Exec Time (ms)      #Exec/sec   CPU Time (ms)   Physical Reads / Exec  #Rows Processed    
              Time                / Exec             (DB Time)    / Exec                                 / Exec
SQL Id        1st  2nd   Diff    1st     2nd         1st  2nd    1st    2nd      1st       2nd          1st  2nd         Multiple Plans   SQL Text
gpgaxqgnssnvt 3.54 15.76 12.23   223,751 1,720,297   0.00 0.00   11,127 49,095   42,333.00 176,565.00   2.67 4.00        No               SELECT DISTINCT VALUE_CURRENCY...