Time Series Storage Design

Hi, I've got the unenviable task of rewriting the data storage back end for a very complex legacy system which analyses time series data for a range of different data sets. What I want to do is bring this data kicking an screaming into the 21st century but putting it into a database. While I have worked with databases for many years I've never really had to put large amounts of data into one and certainly never had to make sure I can get large chunks of that that data very quickly.
The data is shaped like this: multiple data sets (about 10 normally) each with up to 100k rows with each row containing up to 300 data points (grand total of about 300,000,000 data points). In each data set all rows contain the same number of points but not all data sets will contain the same number of points as each other. I will typically need to access a whole data set at a time but I need to be able to address individual points (or at least rows) as well.
My current thinking is that storing each data point separately, while great from a access point of view, probably isn't practical from a speed point of view. Combined with the fact that most operations are performed on a whole row at a time I think row based storage is probably the best option.
Of the row based storage solutions I think I have two options: multiple columns and array based. I'm favouring a single column holding an array of data points as it fits well with the requirement that different data sets can have different numbers of points. If I have separate columns I'm probably into multiple tables for the data and dynamic table / column creation.
To make sure this solution is fast I was thinking of using hibernate with caching turned on. Alternatively I've used JBoss Cache with great results in the past.
Does this sound like a solution that will fly? Have I missed anything obvious? I'm hoping someone might help me check over my thinking before I commit serious amounts of time to this...

Hi,
Time Series Key Figure:
        Basically Time series key figure is used in Demand planning only. Whenever you cerated a key figure & add it to DP planning area then it is automatically convert it in to time series key figure. Whenever you actiavte the planning area that means you activate each Key figure of planning area with time series planning version.
       There is one more type of Key figure & i.e. an order series key figure & which mainly used in to SNP planning area.
Storage Bucket profile:
      SBP is used to create a space in to live cache for the periodicity like from 2003 to 2010 etc. Whenever you create SBP then it will occupy space in the live cache for the respective periodicity & which we can use to planning area to store the data. So storage bucket is used for storing the data of planning area.
Time/Planning bucket profile:
     basically TBP is used to define periodicity in to the data view. If you want to see the data view in the year, Monthly, Weekly & daily bucket that you have to define in to TBP.
Hope this will help you.
Regards
Sujay

Similar Messages

Time-series / temporal database - design advice for DWH/OLAP???

I am in front of task to design some DWH as effectively as it can be - for time series data analysis - are there some special design advices or best practices available? Or can the ordinary DWH/OLAP design concepts be used? I ask this - because I have seen the term 'time series database' in academia literature (but without further references) and also - I have heard the term 'temporal database' (as far as I have heard - it is not just a matter for logging of data changes etc.)
So - it would be very nice if some can give me some hints about this type design problems?

Hi Frank,
Thanks for that - after 8 years of working with Oracle Forms and afterwards the same again with ADF, I still find it hard sometimes when using ADF to understand the best approach to a particular problem - there is so many different ways of doing things/where to put the code/how to call it etc... ! Things seemed so much simplier back in the Forms days !
Chandra - thanks for the information but this doesn't suit my requirements - I originally went down that path thinking/expecting it to be the holy grail but ran into all sorts of problems as it means that the dates are always being converted into users timezone regardless of whether or not they are creating the transaction or viewing an earlier one. I need the correct "date" to be stored in the database when a user creates/updates a record (for example in California) and this needs to be preserved for other users in different timezones. For example, when a management user in London views that record, the date has got to remain the date that the user entered, and not what the date was in London at the time (eg user entered 14th Feb (23:00) - when London user views it, it must still say 14th Feb even though it was the 15th in London at the time). Global settings like you are using in the adf-config file made this difficult. This is why I went back to stripping all timezone settings back out of the ADF application and relied on database session timezones instead - and when displaying a default date to the user, use the timestamp from the database to ensure the users "date" is displayed.
Cheers,
Brent

Time-Series Data storage

What's the most effiecient way to store time-series data in oracle?.
Thanks,
Jay.

937054 wrote:
Hello,
1. Usally time-series data goes in multiple millions, so timeseries databases like FAME,KDB,SYBASE-IQ are used. Does oracle11gr2 provide storage optmizations, compressions, columnar database like FAME,KDB,SYBASE-IQ?
The only methods of optimization are partitioning of the data by some date or if data set is narrow (few columns)l enough, partitioned IOT.
2. http://www.oracle.com/us/corporate/press/1515738
Link is about R statistical language and data mining integration with Oracle database 11gR2. Does this come default during installation or with BigData - EXADATA? OR this is a seperate license?I am not sure about the licensing, you will need to ask your sales person, but it looks like it might be apart of ODM (oracle data mining - a licensed product)
Take a read through this case study.
http://www.oracle.com/technetwork/database/options/advanced-analytics/odm/odmtelcowhitepaper-326595.pdf?ssSourceSiteId=ocomen
>
Thanks

Time series inconsistency due to change in Fiscal variant/Storage bucket

Hi All,
We got into the situation where running time series consistency check on a DP planning area fails with the message pointing to, 'changes to Fiscal variant'. We are unable to track any changes to the fiscal variant though there is a possibility that storage bucket horizon could have been changed. Is there a way to correct/synch this by running a report?
We are trying to avoid re-initialization here, though this is an option, for the obvious reasons to backup the data in production. Is there an alternative solution to fix this inconsistency?
Cheers!

Dear Ashok,
You should never change a FYV when this FYV is used in a storage
bucket profile which is used in a planning area and if time series
objects are already created. It is no problem to maintain the FYV for
an additional time range, but changes and deletion of periods should
never be made if the FYV is actively used. You should not
change existing periods. If you want to make any changes to buckets
in your FYV or delete buckets, you should always make a COMPLETE
backup of ALL your data from the planning area into an InfoCube,
then delete the time series objects for the planning area (and with
this all the data in liveCache) and then change the FYV. After that,
you can create time series objects again and reload the data from the
Backup-InfoCube into the planning area. If you do these steps, you
will not risk to loose any data. The data in the InfoCube will be
the backup you can reload.
As some processes check the FYV some time before and after it is used,
it is very recommendable to maintain the FYV at least 2 - 3 years in
the past and in the future. E.g. if you create time series objects up
from 2001 you should maintain your FYV at least back to 1999 and if you
create time series objects until 2006, you should maintain your FYV
at least until 2008. It might be that you never experience problems if
you do not maintain the FYV in this time range, but some processes do
additional checks and then it can come to problems
Regards,
Rakesh

How to apply time series function on physical columns in OBIEE RPD

Hi,
I know a way to apply time series function(Ago and ToDate) by using existing logical columns as the source. I have set the chronological key and created time dimension. In the expressiion builder for the same Time Dimension appears at the Top so that we can use its level.
But I couldn't apply a time series function when i have to create a logical column using physical columns. In the expression builder for the same, Time dimension does not appear in the list. Neither can i use any column from the time dimension. Please let me know a way to do it.
Thanks.

Time series functions are - by design and purpose - only valid for derived logical columns and not useable inside physical mappings.
If you want / need to do it on a physical level, then abandon the time series functions and do it the old-school way with multiple LTS instances.

Bad performance due to the use of AGO and TO_DATE time series functions

Hi all,
I'm building an OBI EE Project on top of a 1TB DW, and i'm facing major performance problems due to the use of the AGO and TO_DATE time series functions in some of the Metrics included on the reports. I discovered that when a report with one of those metrics is submited to the DB, the resulting query/explain plan is just awful!... Apparently OBI EE is asking the DB to send everything it needs to do the calculations itself. The CPU cost goes to the roof!
I've tried new indexes, updated statistics, MV's, but the result remains the same, i.e., if you happen to use AGO or TO_DATE in the report you'll get lousy query time...
Please advise, if you have come across the same problem.
Thanks in advance.

Nico,
Combining the solution to view the data in dense form (http://gerardnico.com/wiki/dat/obiee/bi_server/design/obiee_densification_design_preservation_dimension), and the use of the lag function (http://gerardnico.com/wiki/dat/obiee/presentation_service/obiee_period_to_period_lag_lead_function) appears to be the best solution for us.
Thanks very much.

Semiweekly Time Series Creation for DP

Hi,
I want to create a semiweekly (twice a week) time series for DP.
And this would be spread for 2 years (i.e. 1 year for the past and 1 year for the future).
So a total of 208 entries need to be made for this time series.
My understanding of the method is
1)     Create a fiscal Year Variant.
2)     Create a Storage Bucket profile based on fiscal Year Variant.
3)     Create a Time Series Object based on Storage Bucket profile.
4)     In the new Planning Area, provide the Time Series Object.
Correct me if my understanding is wrong.
Also can you provide the T-Code or Menu Path for the first three activities?
Regards,
Vikas

The Menu Path for
(1) Fiscal Year Variants
SPRO>APO> Supply CHain PLanning> Demand Planning> Basic Settings>Maintain Fiscal Year Variants
(2) Storage Bucket Profiles
SPRO>APO> Supply CHain PLanning> Demand Planning> Basic Settings>Define STorage Bucket Profiles
(3) I guess when you mean time series, you meant planning bucket profile
SPRO>APO> Supply CHain PLanning> Demand Planning> Basic Settings>Define Planning Bucket Profiles
(4) Define PLanning Area- /n/sapapo/msdp_admin
You define a storage bucket profile
(5) Define PLanning Book
You define the planning bucket profile

YTD, MTD, YAGO, Function (Time-series Functions TODATE, AGO) Like OBIEE

Dear BO Gurus,
I am a newbie to business object. I would like to ask are there any ready made time series functions which can be used in the BO Semantic layer (universe).
Like Year to Date, Year Ago. Functions like TODATE and AGO functions found in OBIEE semantic layer (siebel analytics). It was just simple there, I had to use the function on the semantic layer and BI server would raise the query to get the answer.
The syntax was
Todate(measure, time_dim_attribute)
Ago(measure, Time_Dim_Attribute, period)
Thanks & Regards
Ishaq

Dear Gurus,
There is no DB function in oracle to do this (AGO/TODATE).
OBIEE issues time-series queries, in the form of AGO and TODATE. AGO gives you, for example, the value of sales one month ago or one quarter ago, whilst TODATE gives you the total of sales month-to-date, or quarter-to-date, or year-to-date. Both of these time-series functions rely on the existence of a time dimension in your business model and mapping layer, with the period you can refer to in the AGO and TODATE functions being determined by what levels you have in this time dimension.
This eliminates the need to create and store complex time-based
reporting tables.
I would only like to know does the universe designer has this sort of functionality which is present in OBIEE semantic layer. Or I need to create and store time-based reporting tables.
The question can best be answered by some-one has worked on both the BI platforms (OBIEE and BO).
Thanks & Regards
Ishaq

Time series error

hi friends
got this error
new to 11g
plz can any one help me out of this
State: HY000. Code: 10058. [NQODBC] [SQL_STATE: HY000] [nQSError: 10058] A general error has occurred. [nQSError: 43113] Message returned from OBIS. [nQSError: 22040] To use Ago function, the storage level of the query ('[TIMES.TIME_ID]') must be a static level. (HY000)

>
hi friends
got this error
new to 11g
plz can any one help me out of this
State: HY000. Code: 10058. [NQODBC] [SQL_STATE: HY000] [nQSError: 10058] A general error has occurred. [nQSError: 43113] Message returned from OBIS. [nQSError: 22040] To use Ago function, the storage level of the query ('[TIMES.TIME_ID]') must be a static level. (HY000)
>
Please mark this question ANSWERED and repost it in the OBIS forum as this is question about Siebel Analytics which is now part of Oracle's BI.
https://forums.oracle.com/forums/category.jspa?categoryID=147
That error is because you have not set up your time dimension properly. Did you perhaps use the OBIEE extract() function to derive your dimension?
If so you should create a view or table instead and the error should go away.
See this article on 'OBI EE, Time Dimensions and Time-Series Calculations for an example of how to set up your dimensions properly.
http://www.rittmanmead.com/2007/04/obi-ee-time-dimensions-and-time-series-calculations/

Time Series initialization dates with fiscal periods!

Dear Experts
Problem:
I cannot initialize planning area.
Period YYYY00X is invalid for periodicity P FYV
Configuration
Storage Buckets Profile: Week, Month, Quarter, Year, Post Period FYV all checked.
Horizon
Start: 24.02.2013
End: 31.12.2104
No Time stream defined.
Fiscal Year Variant
2012
Month
1
2
3
4
5
6
7
8
9
10
11
12
Edate
28
25
31
28
26
30
28
25
29
27
24
31
FP
1
2
3
4
5
6
7
8
9
10
11
12
2013
Month
1
2
3
4
5
6
7
8
9
10
11
12
Edate
26
23
30
27
25
29
27
24
28
26
23
31
FP
1
2
3
4
5
6
7
8
9
10
11
12
2014
Month
1
2
3
4
5
6
7
8
9
10
11
12
Edate
25
22
29
26
24
28
26
23
27
25
22
31
FP
1
2
3
4
5
6
7
8
9
10
11
12
2015
Month
1
2
4
5
5
7
8
8
10
10
11
12
Edate
31
28
4
2
30
4
1
29
3
31
28
31
FP
1
2
3
4
5
6
7
8
9
10
11
12
Question
What goddamn dates should I enter in the planning area initialization start and end to initialize for maximum duration given the settings above?
I tried a few dozens but none is accepted. For a start I tried the same dates as in Horizon of storage bucket profile. But given the kind of error text I have I cannot decipher with tiny little mind what dates I am expected to enter in for time series creation.
Thanks
BS

Thanks Mitesh,
No its 2014
Here is what worked.
Storage Bucket Horizon
Start: 24.02.2013
End: 22.11.2014
Time Series Initialization
Start: 01.02.2013
End Date: 31.12.2014
The fiscal year variant is what I pasted above.
I thought time series can only be initialized for a subset of period in the storage bucket profile !. This is my first experience of this kind where Initialization period is larger than the storage horizon. Is this the case ? I went by the book always and this is just a chance discovery by me.
I was aware of the limitation by SAP notes on need to have one more year on either sides in fiscal year variant for it to be used for initialization and hence the range of dates I tried.
Appreciate your comments on this.
All dates in dd/mm/yyyy.
Thanks
BS

Time Series Objects for a Planning Area

Hi all,
Can anyone let me know why do we create Time Series Objects for a Planning Area.
What is its Role & significance..
Regards,.
Vishal.S.Pandya

time series is usually a Demand planning concept(and used in SNP as well but in SNP its predominantly the order series that plays a main role)
time series is a general concept in statistics(and forecasting) wherein the value of a key figure is represented in a time bucket
Time series gives you an idea of the gradual change in the values by time and the relation of the future based on the past
planning area in APO (and other tools) tries to represent this as a 2 dimensional model with time on the columns and key figures in the rows. the value that you load into the cells that are formed by the above are based on the characterisitic values you choose from your MPOS which is linked to the way the values are stored in the planning area
The planning area stores data for each key figure in the smallest unit of time(technical storage buckets) and the lowest level of characterisitc value combination

Balance for open periods in a time series enabled cube

Hi,
We have a request to calculate balance for open periods in a time series enabled cube, I don’t there is any inbuilt function for this.
Any ideas on how to calculate this without adding new members/formulae in the time dimension. Currently we compute this in the report
EX:
BOY(May) = Jun+Jul + … + Dec
Thanks,

You will have to add a member somewhere. Suppose you add it in scenario, then you could use sumrange for the current member range offsetting the current monthe by 1 and use null for the end month wich gives you the last month in the time dim.

Order Series key figure is considered as Time Series

Hi Gurus,
I'm adding a new Key FIgure (Order type) into a copied version of SNP Planning Area.
I have assigned category and therefore category group and, applied similar settings like other KFs (Planning Area, Planning Bokk...). The thing is when going into Data View, it's not behaving as an Order Type and saves its data as a Time Series. Thus no "detailed information" appears when I right-click in the bucket.
Do you have any idea where my settings are wrong?
PS: I'm using SCM 7.0 module version
Many thanks & regards,
David

Hi David,
You defined the keyfigure as order keyfigure.So when enter value in this keyigure system tries to create order as per keyfigure function which is assigned to this keyfigure.please note that not all the keyfigure functions creates the order.
If your keyfigure function is anyone of the following then system creates the stock transaport order:
4001
4015
4002
4007
4006
4013
4018
4019
4020
If your keyfigure function is anyone of the following then system creates planned orders:
2001
9020
2007
If your keyfigure function is anyone of the following then system creates forecast:
1002
1003
1004
Hope this information will helps you.
Regards,
Sunitha

Time-series Chart

Warning, CR Newbie here so this may be a stupid question. I am evaluating the trial version of CR to see if it will be a good fit for an upcoming project. I've seen some related posts in the SCN, but no answers that quite fit.
I'm looking to create a line chart (or a scatter chart) with time-series data. My dataset includes a time stamp field (yyyy-MM-dd hh:mm:ss) and some floating-point temperature values like this:
2014-05-01 08:00:00, 123.4, 115.1, 109.2
2014-05-01 08:00:10, 123.6, 116.0, 109.8
The desired outcome has the date / time along the X-axis with data points spaced proportionally in the X dimension and plotted in the Y-dimension according to the temperature. The interval between the time stamps is not always the same, so numerical scaling is required on both axes. The desired chart would show a temperature scale along the vertical axis, three trend lines for the three series of temperature data and times shown on the X axis label.
I've played with several options in an attempt to make this work. On the data tab, it would seem I would want to select "on change of" and then my time-stamp field. However, with this selection, I can only use summary values and end up with a chart with a single data point for each series. I don't need or want any summary calculations carried out on the data, I just want to plot it so I can look at a trend over time. I can get trend lines if I select "for each record" on the data tab of the wizard, but then my X-axis is meaningless and the horizontal scaling is misleading unless the interval between my samples is constant.
I would welcome any suggestions on how best to accomplish this with Crystal Reports.
Thanks for reading.

Jamie,
Thanks for continuing to reply. I am getting close, but still no success.
Here is the procedure I've followed and problem:
Put chart in RF section
Start Chart Expert
Chart Type = Numeric Axes, subtype = Date axis line chart
Data tab
On change of datetime field
Order... ascending, printed for each second
Values avg of my data fields (must select summary when on change of is used)
Right-click on X-axis label, select Group (X) Axis Settings
Scales tab: base unit, major unit and minor unit can only be set to days, months or years
I cannot set the minimum and maximum date with resolution other than day
Right-click Chart, select Chart Options...Axes tab: show group axes set to show time scale
No matter the setting I use, I can't find a way to adjust the resolution of the time scale lower than days.
I tried using a formula to extract only the time portion of my datetime field. I used that as my "on change" data series, hoping maybe CR would automatically recognize I was looking at a fraction of a day if I did that. No good - now it gives me a date scale with the dates showing up as the beginning of the epoch, but I can still only get resolution of integer days.
Thanks for your patience and persistence.
- Max

TODATE time series function in OBIEE 11g

Hi,
I have a problem with time series function. I have month level in time dimension hierarchy.
I have used below expression to get month to date results in my reports.
column
expression----> TODATE(fact.measure, hierarchy.Month level);
when i am using this column in my reports it is showing null values. The below error i am getting in view log files
----------------> Converted to null because it's grain is below query's grain
Note: Here i have measures, year,qtr,month,day,shift,hour in single physical table in physical layer.
Is it a problem to have measures and time columns in a single physical table?
Please let me know if you have any solution.
Thanks,
Avinash

Yes, it shud be a prob. Try using seperate tables for fact n timedim

Time Series Storage Design

Similar Messages

Maybe you are looking for