Best Partition for Time Series

Question

Hi All,
i have the following tables in my DB
CREATE TABLE READING_DWR (
ID     VARCHAR(20)     NOT NULL,
MACHINE_ID     VARCHAR(20),
DATE_ID     NUMBER,
TIME_ID NUMBER,
READING NUMBER
CREATE TABLE DATE_DIMENSION (
DATE_ID     NUMBER     NOT NULL,
DATE_VALUE     DATE     NOT NULL,
DAY     VARCHAR(10),
DAY_OF_WEEK     INTEGER,
DAY_OF_MONTH     INTEGER,
DAY_OF_YEAR     INTEGER,
PREVIOUS_DAY     DATE,
NEXT_DAY     DATE,
WEEK_OF_YEAR     INTEGER,
MONTH     VARCHAR(10),
MONTH_OF_YEAR     INTEGER,
QUARTER_OF_YEAR     INTEGER,
YEAR     INTEGER
CREATE TABLE TIME_DIMENSION (
TIME_ID     NUMBER     NOT NULL,
HOUR     VARCHAR(3),
MINUTE     VARCHAR(3),
SECOND     VARCHAR(3),
INTERVAL     NUMBER
Referential Constrains:-
STG_READING(DATE_ID)>>>>>DATE_DIMENSION(DATE_ID)
STG_READING(TIME_ID)>>>>>TIME_DIMENSION(TIME_ID)
READING_DWR contains the time series data for a particular machine.
What is the best way to partition the READING_DWR to improve the performance of my select query??

Answer 1

Thanks for posting the additional information. I think I have a better understanding of what you are trying to do.
As I suspected partitioning has nothing to do with it.
>
Now where the first value is null , i have to get the record from the READING_DWR , where the time is less then 10:00 for a particular machIne
>
If I understand what you what you are trying to do correctly it is something like this. Please correct anything that is wrong.
1. READING_DWR is a history table - for each machine_id there is a datetime value and an amount which represents a 'total_to_date' value
2. STG_READING is a stage table - this table has new data that will be (but hasn't been) added to the READING_DWR table. All data in this table has a later datetime value than any data in the READING_DWR table. You know what the date cutoff is for each batch; in your example the earliest date is 10:00
3. You need to report on all records from STG_READING (which has 'total_to_date') and determine the 'incremental-value'; that is, the increase of this value from the preceding value.
4. For the first record (earliest datetime value) in the record set for each machine_id the preceding value will be the value of the READING_DWR table for that machine_id for the record that has the latest datetime value.
5. Your problem is how to best meet the requirement of step 4 above: that is, getting and using the proper record from the READING_DWR table.
If the above is correct then basically you need to optimize the 'getting' since you already posted code that uses the LAG (1 record) function to give you the data you need; you are just missing a record.
So where you show output that was from only the STG table
>
Now the output will be
=======================
Time Reading lag
10:00 200 null
10:15 220 200
10:20 225 220
10:30 230 225
>
If you include the DWR record (and no other changes) the output might look like
>
Time Reading lag
08:23 185 null
10:00 200 185
10:15 220 200
10:20 225 220
10:30 230 225
>
The above output is exactly what you want but without the first record. I assume you already know how to eliminate one record from a result set.
So the process for what you need, in pseudo-code, basically boils down to:
WITH ALL_RECORDS_NEEDED AS (
SELECT machine_id, last_record_data FROM READING_DWR
UNION ALL
SELECT * FROM STG_READING
SELECT lag_query_goes_here FROM ALL_RECORDS_NEEDEDThen either ignore or remove the earliest record for each machine_id since it came from READING_DWR and will have a NULL for the lag value. If you add a flag column to each query to indicate where the data came from (e.g. 'R' for READING_DWR and 'S' for STG_READING) then you can just use the records with a flag of 'S' in a report query or outer query.
So now the problem is reduce to two things:
1. Efficiently finding the records needed from the READING_DWR table
2. Combining the one DWR record with the staging records.
For #1 since you want the latest date for each machine_id then an index COULD help. You said you have an index
>
index on READING_DWR---MACHINE_ID,DATE_ID,TIME_ID
>
But for a query to find the latest date you want DATE_ID and TIME_ID to be in descending order.
The problem here is that you have seriously garbaged up your data by using numbers for dates and times - requiring
>
TO_DATE(DATE_ID||''||LPAD(time_id,6,0),'YYYYMMDDHH24MISS'))
>
to make it useful.
This is a VERY BAD IDEA. If at all possible you should correct it. The best way to do that is to use a DATE column in both tables and convert the data to the proper date values when you insert it.
If that is not possible then you should create a VIRTUAL column using your TO_DATE functionality so that you can index and query the virtual column as if it were a date.
For #2 (Combining the one DWR record with the staging records) you can either just union the two queries together (as in my psuedo-code) or extract a copy of the DWR and insert it into the staging table.
In short query ALL of the DWR records you need (one for each machine_id) separately as a batch and then combine them with the STG records. Don't look them up one at a time like your posted code is trying to do.
If your process is something like this and perhaps run every 15 minutes
1. truncate the stage table
2. run my report
3. add stage records to the history table
Then I would modify the process to use the 15 minutes 'dead' time between batches to extract the DWR records needed for the next batch into a holding table. Once you do step 3 above (update the history table) you can run this query and have the records preprocessed for your next batch and report.
I would use a new holding table for this purpose rather than have the staging table server a double purpose. You never know when you might need to redo the staging table load; this means truncating the table which would wipe out the DWR staged records.
Anyway - with all of the above you should be able to get it working and performing.