Pls. help tune this query

Question

This is the SQL i would like to tune for performace...
The table structure is given below.
The table has about 5 million rows.
On the first day, load_flag has all the rows as 'I'.
Then from the second day onwards only around 10% of records will be between Load_Start_Time and Load_End_Time. Among these around 40% will have Record_key like 'TP%'. And among those most of the records (95%)will have load_flag as 'U' and a very few (5%) as 'I'. At present there are unique and primary key indexes on record_key. Please advice me whether it's better to go for an index on any of these columns and what type would be better. I thought it would help to have a bit map index on load_flag and a function base index on SUBSTR (RECORD_KEY).
Also pls. let me know if the order of predicates is right.
Thanks in advance.
M_STG_TPDB_TPD_TL_W_PH_LOI_CNTBLTY_STATUS_CE     SQ_STG_TPD_STG_TL_CS_EXTRACTED_RECS_MOD     "SELECT TPD_STG_TL_CS_EXTRACTED_RECS.RECORD_KEY, TPD_STG_TL_CS_EXTRACTED_RECS.DATA_SOURCE, TPD_STG_TL_CS_EXTRACTED_RECS.CONTACTABLE_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.LEGAL_OWNERSHIP_ISSUE_IND, TPD_STG_TL_CS_EXTRACTED_RECS.ADMIN_CONTROL_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.BANKRUPTCY_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.ASSIGNED_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.IN_TRUST_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.DIVORCE_CASE_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.POA_COP_INDICATOR, TPD_STG_TL_CS_EXTRACTED_RECS.SOURCE_EXTRACT_DATE_TIME
FROM
TPD_STG_TL_CS_EXTRACTED_RECS
WHERE
LOAD_FLAG IN ('I','U')
AND SUBSTR (RECORD_KEY, 1,2)='TP'
AND STG_UPDATE_DATE_TIME>'$$Load_Start_Time'
AND STG_UPDATE_DATE_TIME<='$$Load_End_Time'"
Table structure
CREATE TABLE TPD_STG_TL_CS_EXTRACTED_RECS
RECORD_KEY VARCHAR2(35 BYTE),
SCHEME_NAME VARCHAR2(50 BYTE),
ORGANISATION_NAME VARCHAR2(50 BYTE),
SUPERIOR_TITLE_1 VARCHAR2(50 BYTE),
TITLE_1 VARCHAR2(50 BYTE),
FIRST_NAME_1 VARCHAR2(50 BYTE),
MIDDLE_NAME_1 VARCHAR2(50 BYTE),
SURNAME_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_1_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_2_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_3_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_4_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_5_1 VARCHAR2(50 BYTE),
ADDRESS_LINE_6_1 VARCHAR2(50 BYTE),
POST_CODE_1 VARCHAR2(12 BYTE),
COUNTRY_1 VARCHAR2(50 BYTE),
OVERSEAS_INDICATOR_1 CHAR(1 BYTE),
DOB_1 NUMBER(8),
GENDER_1 CHAR(1 BYTE),
NINO_1 VARCHAR2(9 BYTE),
DEATH_INDICATOR_1 CHAR(1 BYTE),
PRODUCT_HOLDING_ROLE_TYPE_1 VARCHAR2(21 BYTE),
GONE_AWAY_INDICATOR_1 CHAR(1 BYTE),
THAMES_LEGAL_OWNERSHIP_IND_1 CHAR(1 BYTE),
SOURCE_SYSTEM_PARTY_INDV_ID_1 VARCHAR2(15 BYTE),
SUPERIOR_TITLE_2 VARCHAR2(50 BYTE),
TITLE_2 VARCHAR2(50 BYTE),
FIRST_NAME_2 VARCHAR2(50 BYTE),
MIDDLE_NAME_2 VARCHAR2(50 BYTE),
SURNAME_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_1_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_2_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_3_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_4_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_5_2 VARCHAR2(50 BYTE),
ADDRESS_LINE_6_2 VARCHAR2(50 BYTE),
POST_CODE_2 VARCHAR2(12 BYTE),
COUNTRY_2 VARCHAR2(50 BYTE),
OVERSEAS_INDICATOR_2 CHAR(1 BYTE),
DOB_2 NUMBER(8),
GENDER_2 CHAR(1 BYTE),
NINO_2 VARCHAR2(9 BYTE),
DEATH_INDICATOR_2 CHAR(1 BYTE),
PRODUCT_HOLDING_ROLE_TYPE_2 VARCHAR2(21 BYTE),
GONE_AWAY_INDICATOR_2 CHAR(1 BYTE),
THAMES_LEGAL_OWNERSHIP_IND_2 CHAR(1 BYTE),
SOURCE_SYSTEM_PARTY_INDV_ID_2 VARCHAR2(15 BYTE),
JOINT_OWNER_INDICATOR CHAR(1 BYTE),
JOINT_LIFE_TYPE NUMBER(1),
SAME_ADDRESS_INDICATOR CHAR(1 BYTE),
TITLE_LA1 VARCHAR2(50 BYTE),
FIRST_NAME_LA1 VARCHAR2(50 BYTE),
SURNAME_LA1 VARCHAR2(50 BYTE),
DOB_LA1 NUMBER(8),
TITLE_LA2 VARCHAR2(50 BYTE),
FIRST_NAME_LA2 VARCHAR2(50 BYTE),
SURNAME_LA2 VARCHAR2(50 BYTE),
DOB_LA2 NUMBER(8),
PRODUCT_HOLDING_REF_NUMBER VARCHAR2(28 BYTE),
PARENT_PRODUCT_HOLDING_REF_NUM VARCHAR2(9 BYTE),
OCDB_REFERENCE_NUMBER VARCHAR2(17 BYTE),
BUSINESS_GROUP CHAR(3 BYTE),
SCHEME_NUMBER VARCHAR2(8 BYTE),
TRUSTEE_SEQUENCE_NUMBER NUMBER(10),
MEMBER_NUMBER VARCHAR2(10 BYTE),
PRSN_ID NUMBER(10),
OLD_SCHEME_NUMBER VARCHAR2(8 BYTE),
PUBLIC_SECTOR_INDICATOR CHAR(1 BYTE),
ELIGIBLE_INDICATOR CHAR(1 BYTE),
SCHEME_STATUS NUMBER(1),
SCHEME_TYPE CHAR(2 BYTE),
Q_DATE_WITH_PROFIT_STATUS NUMBER(1),
A_DATE_WITH_PROFIT_STATUS NUMBER(1),
LATEST_WITH_PROFIT_STATUS NUMBER(1),
NPSW_INDICATOR CHAR(1 BYTE),
PRODUCT_HOLDING_STATUS CHAR(1 BYTE),
MATURITY_DATE_OF_CONTRACT NUMBER(8),
DUE_END_DATE_OF_CONTRACT NUMBER(8),
OUT_OF_FORCE_DATE NUMBER(8),
OUT_OF_FORCE_REASON_CODE NUMBER(2),
DATA_SOURCE VARCHAR2(3 BYTE),
PRODUCT_TYPE VARCHAR2(30 BYTE),
PRODUCT_DESCRIPTION VARCHAR2(50 BYTE),
SERVICING_AGENT_NUMBER VARCHAR2(10 BYTE),
CONTACTABLE_INDICATOR CHAR(1 BYTE),
LEGAL_OWNERSHIP_ISSUE_IND CHAR(1 BYTE),
ADMIN_CONTROL_INDICATOR CHAR(1 BYTE),
BANKRUPTCY_INDICATOR CHAR(1 BYTE),
ASSIGNED_INDICATOR CHAR(1 BYTE),
IN_TRUST_INDICATOR CHAR(1 BYTE),
DIVORCE_CASE_INDICATOR CHAR(1 BYTE),
POA_COP_INDICATOR CHAR(1 BYTE),
DONOR_POLICY_INDICATOR CHAR(1 BYTE),
TAX_JURISDICTION NUMBER(2),
INELIGIBLE_DATE NUMBER(8),
INELIGIBILITY_REASON NUMBER(2),
A_DATE_CASH_OR_BONUS_TYPE CHAR(1 BYTE),
VALUATION_APPLICABLE_DATE NUMBER(8),
A_DATE_PIP_AMOUNT NUMBER(12),
A_DATE_NPSW_AMOUNT NUMBER(10),
A_DATE_MINIMUM_APPLIED_AMOUNT NUMBER(10),
A_DATE_MINIMUM_CALC_AMOUNT NUMBER(10),
A_DATE_POLICY_VALUE NUMBER(16),
A_DATE_DATE_PIP_CALCULATED NUMBER(8),
A_DATE_ALGORITHM_NUMBER NUMBER(2),
M_DATE_VOTING_VALUE NUMBER(12),
M_DATE_POLICY_VALUE NUMBER(16),
E_DATE_CASH_OR_BONUS_TYPE CHAR(1 BYTE),
E_DATE_PIP_AMOUNT NUMBER(12),
E_DATE_NPSW_AMOUNT NUMBER(10),
E_DATE_MINIMUM_APPLIED_AMOUNT NUMBER(10),
E_DATE_MINIMUM_CALC_AMOUNT NUMBER(10),
E_DATE_POLICY_VALUE NUMBER(16),
E_DATE_DATE_PIP_CALCULATED NUMBER(8),
E_DATE_ALGORITHM_NUMBER NUMBER(2),
P_DATE_CASH_OR_BONUS_TYPE CHAR(1 BYTE),
P_DATE_PIP_AMOUNT NUMBER(12),
P_DATE_NPSW_AMOUNT NUMBER(10),
P_DATE_MINIMUM_APPLIED_AMOUNT NUMBER(10),
P_DATE_MINIMUM_CALC_AMOUNT NUMBER(10),
P_DATE_POLICY_VALUE NUMBER(16),
P_DATE_DATE_PIP_CALCULATED NUMBER(8),
P_DATE_ALGORITHM_NUMBER NUMBER(2),
SOURCE_EXTRACT_DATE_TIME DATE,
SCHEME_SEQUENCE_NUMBER NUMBER(3),
LOAD_FLAG CHAR(1 BYTE),
STG_CREATE_DATE_TIME DATE,
STG_UPDATE_DATE_TIME DATE
TABLESPACE TPDBS01A_DATA
PCTUSED 0
PCTFREE 10
INITRANS 1
MAXTRANS 255
STORAGE (
INITIAL 128K
NEXT 128K
MINEXTENTS 1
MAXEXTENTS 2147483645
PCTINCREASE 0
BUFFER_POOL DEFAULT
LOGGING
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING;
CREATE UNIQUE INDEX PK_STG_TL_CS_EXTRACTED_RECS ON TPD_STG_TL_CS_EXTRACTED_RECS
(RECORD_KEY)
LOGGING
TABLESPACE TPDBS01A_DATA
PCTFREE 10
INITRANS 2
MAXTRANS 255
STORAGE (
INITIAL 128K
NEXT 128K
MINEXTENTS 1
MAXEXTENTS 2147483645
PCTINCREASE 0
BUFFER_POOL DEFAULT
NOPARALLEL;
ALTER TABLE TPD_STG_TL_CS_EXTRACTED_RECS ADD (
CONSTRAINT PK_STG_TL_CS_EXTRACTED_RECS
PRIMARY KEY
(RECORD_KEY)
USING INDEX
TABLESPACE TPDBS01A_DATA
PCTFREE 10
INITRANS 2
MAXTRANS 255
STORAGE (
INITIAL 128K
NEXT 128K
MINEXTENTS 1
MAXEXTENTS 2147483645
PCTINCREASE 0
));

Answer 1

I would like to discuss a bit more how a composite index is used. This is not anymore the question of the OP, but I think it will deepen my understanding and maybe that of others as well.
So we have this situation to start with.
* A select on two different columns.
* An index on each column would lead to an INDEX RANGE SCAN because of the where condition.
* Instead of 2 indexes where only one would be used (not considering bitmap conversions), we add a composite index on both columns.
* The CBO will choose and access the index.
Question is: How is this access is done in detail?
So 400 rows come out of the index range scan.
This would not be possible if it only scanned one of the two predicates. I agree. The output of the index seem to be only those (400) rows that fit with the where clause on both columns.
This is supported by the fact that an Index only access is possible when we select only information that is in the index (Oracle 9i output).
SQL> explain plan for
  2  select record_key
  3  from mytable
  4  where stg_update_date_time >= to_date('2007-11-11','yyyy-mm-dd')
  5  and stg_update_date_time < to_date('2007-11-12','yyyy-mm-dd')
  6  and record_key like 'TP%'
  7  /
Explained.
SQL>  select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
| Id  | Operation            |  Name       | Rows  | Bytes | Cost  |
|   0 | SELECT STATEMENT     |             |   160 |  2400 |     2 |
|   1 |  INDEX RANGE SCAN    | I1          |   160 |  2400 |     2 |
Note: cpu costing is off, PLAN_TABLE' is old version
9 rows selected.
SQL> Unfortunately I don't see the filter operation, probably because the plan table is old in my system.
The index is build on columns "stg_update_date_time" and "record_key". So it must access these columns in that order.
I think, that maybe the range scan is done on the date column and then a further access/filter operation is done to rule out any index entries (leaf nodes) that do not fit with the LIKE 'TP%' expression.
Since I expect that is is faster to access the data in the index than in the table this should improve performance quite a bit. It is not the same access plan/speed however as with a clause like this:
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
| Id  | Operation                   |  Name       | Rows  | Bytes | Cost  |
|   0 | SELECT STATEMENT            |             |     1 |   520 |     3 |
|   1 |  TABLE ACCESS BY INDEX ROWID| MYTABLE     |     1 |   520 |     3 |
|   2 |   INDEX RANGE SCAN          | I1          |     1 |       |     2 |
Note: cpu costing is off, PLAN_TABLE' is old version
10 rows selected.
SQL> The difference can be seen in the cost estimation.
I hope somebody does understand what I would like to point at.
Result of this considerations:
1) Composite index can be used
2) The index is used as a range scan on one column and as a filter operation after this access on the second column. This is like an extra select statement inside the index structure.
3) The resulting table access is as small as possible.
4) A faster access plan can be achieved when the first column is accessed with direct (unique) scan on the first and a range scan on the second column.
br Sven
Message was edited by:
Sven Weller