Best design for historical data

I'm searching a way to design some historical data.
Here my case:
I have millions of rows defining a population, each on 20-30 different tables.
Those rows are related to one master subject area, let say it is a person.
The details table are defining status in a period of time (marital status, address, gender (yes that may change in some cases!, program, phase, status, support, healthcare, etc). They have all two attributes that define the time period (dateFrom, dateTo), and one or many attributes that define the status of the person (the measure).
I know we need a weekly situation for this population since 1998 to yet.
Problems are those:
1) the population we will analyze will count some 20 different criteria's (measures), may be we will divide those in some different datamarts to avoid to much complexity
2) we will drill down from year to week, and will accomplish comparison like year to year, year to previous year, year to date, etc at each level of time hierarchy (year, quarter, month, week).
The question is :
1) do we need to transform our data at a week level for each person to determine the status at this level (the data may be updated and will cause this transformation to be refreshed) ?
1) do we need to aggregate for each level because mixed situation exists due to fact that more than one status may exists for same person at a given time period (more than one status may exists in march by example, how to interpret it ?), that will cause to some logic to be applied, isn't ?
We will be glad to hear some recommendation/idea about this !
Thank You

I would try to get some exact user requirements, with the dataset you have described there are millions of combinations of answers that can be defined and it will be difficult for you to fulfill them all in one place, especially if there are slowly changing dimensions involved.
The user requirements will determine what levels you transform the data at.

Similar Messages

Best Design for Master data

hi,
I would like to get your suggestions on the data modelling:
I have material, mat_plant and mat_sales.I need to create reports on these master data. But when i created a infoset on those 3 master data, one of my key figure "amount" is becoming 10 times like 1 line for material 10 lines for mat_plant and as a result the report is showing the key figure values 10 times of original value.
Please advise what would be the best way to go...
Thanks in advance...

Adi,
You will get 10 lines only. I guess you have 10 plants for that perticular Plants or sales.
It should come reporting requirements. Please let us know what do you want to report?
relation between 3 master data objects is... 1:N:N (0Material:0MAT_PLANT:0mat_sales).
You can't avoid it... restrict by plant....
Nagesh Ganisetti.

Best practice for heirachical data

First off, I have to say that JMX in Java 6 is terrific stuff. Bundling jconsole in with Java has made JMX adoption so much easier for us.
Now, to my question. We have read-only hierarchical data (think a DOM tree) that we would like to publish via JMX. What is the best practice? We see two possibilities:
1. Publish each node of the tree with it's own object name and type. This will allow jconsole to display the information in the tree control.
2. Publish just the root of the tree with an object name and type and then use CompositeType to describe the nodes of the tree. This means you look at the tree in the "Attribute Value" panel of jconsole.
Is there any best practices for such data? We have implemented #2 and it works but we are wondering if long term this might lead to unforeseen consequences.
Thanks in advance.
--Marty

Path,
I did go with #1 and it worked out great. Every node in our tree is an ObjectName node. Works very well for us.
--Marty

Best practice for sharing data with model window

Hi team,
what would the best practice for sharing data with a modal
window be ? I use a modal window to display record details from a
record list, but i am not quite sure how to access the data from
the components in the main application in the modal window.
Any hints would be welcome
Best
Frank

Pass a reference to the parent into the modal popup. Then you
can reference anything in the parent scope.
I haven't done this i 2.0 yet so I can't give you code. I'll
post if I do.
Oh, also, you can reference the parent using parentDocument.
So in the popup you could do:
parentDocument.myPublicVariable = "whatever";
Tracy

Best choice for a date column in forms

Hello,
I want to know what is the best choice for a date field that the user have to enter manually. I read on a other post that a calender control is not directly implemented in forms 9i. So what is the best choice to be "user friendly" with a date field in a form?
thanks in advance.

Hi
Don't know if this would be much use to you, but I wrote this function 'Check_Date' a while back to check whether the user had typed something which could be interpreted as a date. You'd make your date field into a Text Item, and include a call to this function (which could be in your form, or in a library) in the WHEN-VALIDATE-ITEM trigger on the item, ie:
IF NOT check_date(:ITEMS.date_item1, NULL, NULL) THEN
 RAISE FORM_TRIGGER_FAILURE;
END IF;This would mean that the user could type 'T' or '*' for today, '+1' for tomorrow, '-7' for one week ago, or the date in a variety of formats. When they leave the field, the function tries to resolve what they've typed into a date, and sets the field to a date in the correct format.
('Display_Alert' is just a function to display a named alert with a message.) It can also be used to check that the date entered doesn't fall outside upper and lower bounds. Code follows:
FUNCTION Check_Date (text_in IN OUT VARCHAR2,
 lower_bound_in IN DATE,
 upper_bound_in IN DATE)
RETURN BOOLEAN IS
first_letter VARCHAR2(1) := SUBSTR(text_in, 1, 1);
date_ok BOOLEAN;
this_date DATE;
date_diff NUMBER(8);
output_format VARCHAR2(12) := Get_Application_Property(BUILTIN_DATE_FORMAT);
TYPE date_format_table_type IS TABLE OF VARCHAR2(12)
INDEX BY BINARY_INTEGER;
date_format_table date_format_table_type;
BEGIN
 --Set up table of date formats in order they'll be tested for
 date_format_table(1) := 'DD-MON-RRRR';
 date_format_table(2) := 'DD-MM-RRRR';
 date_format_table(3) := 'DD-MM';
 date_format_table(4) := 'DD-MON';
 date_format_table(5) := 'YYYY-MM-DD';
 date_format_table(6) := 'YYYY-MON-DD';
 --Use of T, or * for today's date...
 IF UPPER(text_in) IN ('T', '*') THEN
 this_date := SYSDATE;
 date_ok := TRUE;
 --Look for use of +x/-x for number of days different from today
 ELSIF first_letter IN ('+', '-') THEN
 BEGIN
 date_diff := TO_NUMBER(SUBSTR(text_in, 2));
 IF first_letter = '+' THEN
 this_date := SYSDATE + date_diff;
 ELSE
 this_date := SYSDATE - date_diff;
 END IF;
 date_ok := TRUE;
 EXCEPTION
 WHEN VALUE_ERROR THEN
 --User has entered something like '+X', so not valid
 date_ok := FALSE;
 END;
 ELSE
 --Go through all the possible formats for date entry,
 --if any give a valid date, then return that.
 FOR ix IN date_format_table.FIRST..date_format_table.LAST LOOP
 EXIT WHEN date_ok;
 BEGIN
 --Try creating a date using the format masks in the local
 --table. If no good, this will throw an INVALID_DATE exception
 this_date := TO_DATE(text_in, date_format_table(ix));
 date_ok := TRUE;
 EXCEPTION
 WHEN OTHERS THEN
 date_ok := FALSE;
 END;
 END LOOP;
 END IF;
 IF date_ok THEN
 --Check for violation of lower/upper bounds on date
 IF (lower_bound_in IS NOT NULL AND this_date < lower_bound_in) THEN
 Display_Alert('info_alert', 'Date cannot be before '||TO_CHAR(lower_bound_in, output_format));
 date_ok := FALSE;
 ELSIF (upper_bound_in IS NOT NULL AND this_date > upper_bound_in) THEN
 Display_Alert('info_alert', 'Date cannot be after '||TO_CHAR(upper_bound_in, output_format));
 date_ok := FALSE;
 ELSE
 text_in := TO_CHAR(this_date, output_format);
 END IF;
 ELSE
 Display_Alert('INFO_ALERT', text_in||' is not a valid date !!');
 END IF;
 RETURN date_ok;
END;Hope this is useful.
regards
Andrew
UK

How to Flip the Sign using Calculation script for historical data

I am currently using Essbase 9.3.1. Its required to flip sign for a specific set of accounts for that i am currently using UDA's to flip the sign.
But now i need to flip the sign for the historical data too. Is there any possible way to flip the sign using calculation scripts for historical data. Kindly let me know your suggestions.
Many thanks in Advance...
Edited by: [email protected] on Jul 30, 2009 9:37 PM
Edited by: [email protected] on Jul 30, 2009 10:58 PM
Edited by: [email protected] on Jul 30, 2009 11:01 PM

Of course there is. This is the kind of calc script that only gets run once, so make usre you test it well before doing it on production and make a backup of production before doing it.
It would be something like
Fix(time frame, accounts to be flipped, level zero other dimensions)
actual = actual * -1;
EndFix
Cal dim dimensions
Note, I chose actual, but you could do it for any dimension that has a single or only a couple of members. What ever dimension you choose to do the calculation on, it can't be included in the fix statement

Sliding window for historical data purge in multiple related tables

All,
It is a well known question of how to efficiently BACKUP and PURGE historical data based on a sliding window.
I have a group of tables, they all have to be backed up and purged based on a sliding time window. These tables have FKs related to each other and these FKs are not necessary the timestamp column. I am considering using partition based on the timestamp column for all these tables, so i can export those out of date partitions and then drop them. The price I have to pay by this design is that timestamp column is actually duplicated many times among parent table, child tables, grand-child tables although the value is the same, but I have to do the partition based on this column in all tables.
It's very much alike the statspack tables, one stats$snapshot and many child tables to store actual statistic data. I am just wondering how statspack.purge does this, since using DELETE statement is very inefficient and time consuming. In statspack tables, snap_time is only stored in stats$snapshot table, not everywhere in it's child table, and they are not partitioned. I guess the procedure is using DELETE statement.
Any thought on other good design options? Or how would you optimize statspack tables historical data backup and purge? Thanks!

hey oracle gurus, any thoughts?

Design issue - Historical data - referencial integrity

Hi,
I dont know if this is the right place to post this question, but
I would to like know about the design issues in storing historical data .
I have a historical table about events ( notifications, alarms, etc ).
In historical table would I store fields with foreign key to the master table ( device table ) ?
In this design , how handle then updates ( remove record ) to the master table ?
would I have a trigger to update the historical table too ? Or I mustn' t permit
updates in master table primary Keys ?
And about no store The foreign keys in the historical data ?
What are the disvantages of this model ?
Thank you,
Faria

Faria,
The answer depends on why you want historical data.
Is the historical data to be used for auditing or legal purposes?
Are you trying to move historical data for performance reasons?
Is there some other purpose?
If you're doing this for auditing purposes, I don't recommend using foreign keys since you won't easily be able to capture delete activity.
I mostly use Designer for all of our designs and typically will turn on server side journaling for this purpose. Designer will then generate the journaling tables and write the triggers to manage the whole process. You don't even need to be a programmer to figure it out.
Let me know back if this isn't your purpose or you need clarification.
Thanks, George

Is this the best design for asynchronous notifications (such as email)? Current design uses Web Site, Azure Service Bus Queue, Table Storage and Cloud Service Worker Role.

I am asking for feedback on this design. Here is an example user story:
As a group admin on the website I want to be notified when a user in my group uploads a file to the group.
Easiest solution would be that in the code handling the upload, we just directly create an email message in there and send it. However, this seems like it isn't really the appropriate level of separation of concerns, so instead we are thinking to have a separate
worker process which does nothing but send notifications. So, the website in the upload code handles receiving the file, extracting some metadata from it (like filename) and writing this to the database. As soon as it is done handling the file upload it then
does two things: Writes the details of the notification to be sent (such as subject, filename, etc...) to a dedicated "notification" table and also creates a message in a queue which the notification sending worker process monitors. The entire sequence
is shown in the diagram below.
My questions are: Do you see any drawbacks in this design? Is there a better design? The team wants to use Azure Worker Roles, Queues and Table storage. Is it the right call to use these components or is this design unnecessarily complex? Quality attribute
requirements are that it is easy to code, easy to maintain, easy to debug at runtime, auditable (history is available of when notifications were sent, etc...), monitor-able. Any other quality attributes you think we should be designing for?
More info:
We are creating a cloud application (in Azure) in which there are at least 2 components. The first is the "source" component (for example a UI / website) in which some action happens or some condition is met that triggers a second component or "worker"
to perform some job. These jobs have details or metadata associated with them which we plan to store in Azure Table Storage. Here is the pattern we are considering:
Steps:
Condition for job met.
Source writes job details to table.
Source puts job in queue.
Asynchronously:
Worker accepts job from queue.
Worker Records DateTimeStarted in table.
Queue marks job marked as "in progress".
Worker performs job.
Worker updates table with details (including DateTimeCompleted).
Worker reports completion to queue.
Job deleted from queue.
Please comment and let me know if I have this right, or if there is some better pattern. For example sake, consider the work to be "sending a notification" such as an email whose template fields are filled from the "details" mentioned in
the pattern.

Hi,
Thanks for your posting.
This development mode can exclude some errors, such as the file upload complete at the same time... from my experience, this is a good choice to achieve the goal.
Best Regards,
Jambor
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Best Practice for Master Data Reporting

Dear SAP-Experts,
We face a challenge at the moment and we are still trying to find the right approach to it:
Business requirement is to analyze SAP Material-related Master Data with the BEx Analyzer (Master Data Reporting)
Questions they want to answer here are for example:
- How many active Materials/SKUs do we have?
- Which country/Sales Org has adopted certain Materials?
- How many Series do we have?
- How many SKUs below to a specific season
- How many SKUs are in a certain product lifecycle
- etc.
The challenge is, that the Master Data is stored in tables with different keys in the R/3.
The keys in these tables are on various levels (a selection below):
- Material
- Material / Sales Org / Distribution Channel
- Material / Grid Value
- Material / Grid Value / Sales Org / Distribution Channel
- Material / Grid Value / Sales Org / Distribution Channel / Season
- Material / Plant
- Material / Plant / Category
- Material / Sales Org / Category
etc.
So even though the information is available on different detail levels, the business requirement is to have one query/report that combines all the information. We are currently struggeling a bit on deciding, what would be the best approach for this requirement. Did anyone face such a requirement before - and what would be the best practice. We already tried to find any information online, but it seems Master data reporting is not very well documented. Thanks a lot for your valuable contribution to this discussion.
Best regards
Lukas

Pass a reference to the parent into the modal popup. Then you
can reference anything in the parent scope.
I haven't done this i 2.0 yet so I can't give you code. I'll
post if I do.
Oh, also, you can reference the parent using parentDocument.
So in the popup you could do:
parentDocument.myPublicVariable = "whatever";
Tracy

Best Design : for a SOAP -XI - BAPI ( Multiple )

Hi all,
Need your suggestion on best design to implement a SOAP -XI - BAPI ( Multiple calls ) scenario.
NON SAP Application will make a SOAP service call to XI. This service need to call 2 BAPI's ( 1st call to get some reqd params for second BAPI call ). Also secon call need a commit.
Whats the best design :
1. ABAP server proxy
2. BPM ( to handle multiple calls )
Thanks and cheers,
Anisha.

>>>>>( 1st call to get some reqd params for second BAPI call ). Also secon call need a commit.
1. ABAP server proxy
Here u can have explicit coding and have better control over the process interms u will call the first bapi and the result of it will be passed to exporting parameter of the second bapi as required.
2. BPM ( to handle multiple calls )
performance hinders as pointed out.

Best method for passing data between nested components

I have a fairly good sized Flex application (if it was
stuffed all into one file--which it used to be--it would be about
3-4k lines of code). I have since started breaking it up into
components and abstracting logic to make it easier to write,
manage, and develop.
The biggest thing that I'm running into is figuring out a way
to pass data between components. Now, I know how to write and use
custom events, so that you dispatch events up the chain of
components, but it seems like that only works one way (bottom-up).
I also know how to make public variables/functions inside the
component and then the caller can just assign that variable or call
that function.
Let's say that I have the following chain of components:
Component A
--Component B
-- -- Component C
-- -- -- Component D
What is the best way to pass data between A and D (in both
directions)?
If I use an event to pass from D to A, it seems as though I
have to write event code in each of the components and do the
bubbling up manually. What I'm really stuck on though, is how to
get data from A to D.
I have a remote object in Component A that goes out and gets
some data from the server, and most all of the other components all
rely on whatever was returned -- so what is the best way to be able
to "share" data between all components? I don't want to have to
pass a variable through B and C just so that D can get it, but I
also don't want to make D go and request the information itself. B
and C might not need the data, so it seems stupid to have to make
it be aware of it.
Any ideas? I hope that my explanation is clear enough...
Thanks.
-Jake

Peter (or anyone else)...
To take this example to the next (albeit parallel) level, how
would you go about creating a class that will let you just
capture/dispatch local data changes? Following along my original
example (Components A-D),let's say that we have this component
architecture:
Component A
--Component B
-- -- Component C
-- -- -- Component D
-- -- Component E
-- -- Comonnent F
How would we go about creating a dispatch scheme for getting
data between Component C and E/F? Maybe in Component C the user
picks a username from a combo box. That selection will drive some
changes in Component E (like triggering a new screen to appear
based on the user). There are no remote methods at play with this
example, just a simple update of a username that's all contained
within the Flex app.
I tried mimicking the technique that we used for the
RemoteObject methods, but things are a bit different this time
around because we're not making a trip to the server. I just want
to be able to register Component E to listen for an event that
would indicate that some data has changed.
Now, once again, I know that I can bubble that information up
to A and then back down to E, but that's sloppy... There has to be
a similar approach to broadcasting events across the entire
application, right?
Here's what I started to come up with so far:
[Event(name="selectUsername", type="CustomEvent")]
public class LocalData extends EventDispatcher
private static var _self:LocalData;
// Constructor
public function LocalData() {
// ?? does anything go here ??
// Returns the singleton instance of this class.
public static function getInstance():LocalData {
if( _self == null ) {
_self = new LocalData();
return _self;
// public method that can be called to dispatch the event.
public static function selectUsername(userObj:Object):void {
dispatchEvent(new CustomEvent(userObj, "selectUsername"));
Then, in the component that wants to dispatch the event, we
do this:
LocalData.selectUsername([some object]);
And in the component that wants to listen for the event:
LocalData.getInstance().addEventListener("selectUsername",
selectUsername_Result);
public function selectUsername_Result(e:CustomEvent):void {
// handle results here
The problem with this is that when I go to compile it, it
doesn't like my use of "dispatchEvent" inside that public static
method. Tells me, "Call to possibly undefined method
"dispatchEvent". Huh? Why would it be undefined?
Does it make sense with where I'm going?
Any help is greatly appreciated.
Thanks!
-Jacob

Best Practise for loading data into BW CSV vs XML ?

Hi Everyone,
I would like to get some of your thoughts on what file format would be best or most efficient to push data into BW. CSV or XML ?
Also what are the advantages / Disadvantages?
Appreciate your thoughts.

XML is used only for small data fields - more like it is easier to do it by XML rather than build an application for the same - provided the usage is less.
Flat files are used for HUGE data loads ( non SAP ) and definitely the choice of data formats would be flat files.
Also XML files are transformed into a flat file type format with each tag referring to the field and the size of the XML file grows to a high value depending on the number of fields.
Arun

Best practices for accessing data in subviews

I've got two iPhone projects which share most of their code base. I'm trying to figure out the best way to load data from some plist files and store them in a common container UIView and provide access to the data for subviews and subviews of the subviews, etc. Right now I've got the data being passed from the container view to the subviews it creates and then the subview themselves pass it further, basically a bucket brigade to get the data to where it needs to go which could be 3 or 4 views down in the hierarchy.
Is there a better approach? I've looked at delegates & protocols but I'm having a hard time understanding how they work and whether they are appropriate in this situation. Originally I had the app delegate holding the data and any class anywhere could invoke the app delegate and get the data. However this approach fails with 2 projects because the app delegates have different names and the classes that need to access it are common to both projects. Can the app delegate be renamed without significant impact? Or is there a way a UIView can be set up as a delegate in much the same way?
Thanks for any advice!
Greg

Hi Greg - You can rename the app delegate class to whatever you want, just remember to change it in IB, and make sure you catch any place it already appears in your code. I guess there's no need to change the app delegate class file names unless you want to.
However there are lots of other solutions to your problem. A case could be made for declaring a global pointer to this data, for example. Or, you could encapsulate the data wherever you want and make an extern (globally visible) C function to access it.
Another solution would be to put the data in a shared object which would be accessed just like the shared app object, e.g.
#include "MyObject.h"
NSDictionary *myPlistData = [MyObject sharedObject].plistData;
I just got done looking at "how to make a shared object" in the Cocoa docs and can't seem to find it atm. Anyway it's in there somewhere, either in an Obj-C doc or one of the top level guides. The job just wants a class method that returns the pointer stored in a static C var; if the var is nil, the object is first created and its addy is stored in the var.
Hope that helps!
- Ray

Data Model best Practices for Large Data Models

We are currently rolling out Hyperion IR 11.1.x and are trying to establish best practces for BQYs and how to display these models to our end users.
So far, we have created an OCE file that limits the selectable tables to only those that are within the model.
Then, we created a BQY that brings in the tables to a data model, created metatopics for the main tables and integrated the descriptions via lookups in the meta topics.
This seems to be ok, however, anytime I try to add items to a query, as soon as i add columns from different tables, the app freezes up, hogs a bunch of memory and then closes itself.
Obviously, this isnt' acceptable to be given to our end users like this, so i'm asking for suggestions.
Are there settings I can change to get around this memory sucking issue? Do I need to use a smaller model?
and in general, how are you all deploying this tool to your users? Our users are accustomed to a pre built data model so they can just click and add the fields they want and hit submit. How do I get close to that ideal with this tool?
thanks for any help/advice.

I answered my own question. in the case of the large data model, the tool by default was attempting to calculate every possible join path to get from Table A to Table B (even though there is a direct join between them).
in the data model options, I changed the join setting to use the join path with the least number of topics. This skipped the extraneous steps and allowed me to proceed as normal.
hope this helps anyone else who may bump into this issue.

Best design for historical data

Similar Messages

Maybe you are looking for