A robust data structure for a histogram of events over time.
Hi all,
I've been thinking for days, I am still lost on how to solve the following:
Let's say that we have a list of events that occur over a certain time interval. Each event has a start timestamp and a finish timestamp. The events occur in no particular order and expand over various length of periods.
With respect to the histogram, we mark the time period of an event occuring with a frequency of one. When events overlap over a time period, we mark that time period with a frequency of number of overlapping events.
Given this raw data, we want to examine the data's frequency under various granularity such as second, minutes, hours, days, weeks, month and years for 60sec, 60min, 24 hrs, 7 days, 4 wks ranges respectively.
So, what is a good intermediate (general) data structure that can store all the various events such that they can be easily transformed to a specific granularity. My goal here is to read the event list once, and histogram of events data with respect to various timeframe can be calculated.
Does a timeline like data strucuture that stores aggregated frequencies solve my problem? With a fine granuarlity say seconds and a long time range, the data structure will bound to overflow.
Any help is greatly appreciated!
Brian
You data structure would be in terms of Classes. You need a class Anevent{
String eventName;
Date startTime;
Date endTime;
In your main program have a Vector object which can store unllimited objects and which would not overflow. Store your event objects as they happen in this Vector. After your time interval of events is completed and you have the final Vector, you can pass the vector object to various methods in your main program to be simple or pass it to specailized graph drawing classes you would have to write to draw various Histograms.
Similar Messages
-
Data structure for web-based discussion forum
Hello all. I have begun work on a discussion board forum but have not yet decided upon a data structure for the hierarchy of messages to be left on a board. Any one have any thoughts or experience in this matter? My first thought was to implement a basic linked list in which every reply to a message would be a new child of a node. This would require I use recursion in some areas for counting and finding messages but I was also thinking that this way I can store each thread object in a simple fashion using serialization.
SJTHE STANDARS DIRECTORY STRUCTURE IS :-
PROJECT FOLDER
|
|---------->WEB-INF
|------> web.xml file
|-------> classes (folder to keep java files)
|-------> lib (folder to keep jar files) -
Looking for suitable and robust data structure?
I have an application where i have to give user an option to correlate the
response returned by a method call with other methods arguments.
I have provided a jtree struture where user can see methods and their
responses and can correlate them by selecting appropriate arguments
where these values could be used.
I used a data structure to store such relations like
A hashmap where every key is a unique number representing method number
and value ia a matrix.
In each matrix, the first element of every row would be the Response object and
all remaining elments in the row would be Argument objects. :)
Now maybe someone would ask, for every method there is always only one return type
then why such a matrix with Reponse object as first element of each row.
The reason is that i gave user an option to not only relate response as a whole Object to
different arguments of different methods but user can select a field of this response object if this
is a complex type or user can select any element if the response is a Collection object to relate
with arguments.
I am not sure whether this structure is more appropriate or i can use more robust
and memory efficient structure.
Please give me your useful suggestions.J2EER wrote:
I have an application where i have to give user an option to correlate the
response returned by a method call with other methods arguments.I really don't understand you.
I have provided a jtree struture where user can see methods and their
responses and can correlate them by selecting appropriate arguments
where these values could be used.Huh?
I used a data structure to store such relations likeErr, what happened to the rest of your sentence? Or is the next part the rest of it?
A hashmap where every key is a unique number representing method number
and value ia a matrix.Where does this matrix come from? In my book, a matrix is a 2 dimensional structure. What do you mean by it?
In each matrix, the first element of every row would be the Response object and
all remaining elments in the row would be Argument objects. :)
Now maybe someone would ask, for every method there is always only one return type
then why such a matrix with Reponse object as first element of each row.You're still talking about methods? If so, don't all methods have just one return type?
The reason is that i gave user an option to not only relate response as a whole Object to
different arguments of different methods but user can select a field of this response object if this
is a complex type or user can select any element if the response is a Collection object to relate
with arguments.
I am not sure whether this structure is more appropriate or i can use more robust
and memory efficient structure.
Please give me your useful suggestions.Perhaps you could give a more detailed explanation and more importantly, provide some examples of what you mean. -
What is the best data structure for loading an enterprise Power BI site?
Hi folks, I'd sure appreciate some help here!
I'm a kinda old-fashioned gal and a bit of a traditionalist, building enterprise data warehouses out of Analysis Service hypercubes with a whole raft of MDX for analytics. Those puppies would sit up and beg when you asked them to deliver up goodies
to SSRS or PowerView.
But Power BI is a whole new game for me.
Should I be exposing each dimension and fact table in the relational data warehouse as a single Odata feed?
Should I be running Data Management Gateway and exposing each table in my RDW individually?
Should I be flattening my stars and snowflakes and creating a very wide First Normal Form dataset with everything relating to each fact?
I guess my real question, folks, is what's the optimum way of exposing data to the Power BI cloud?
And my subsidiary question is this: am I right in saying that all the data management, validation, cleansing, and regular ETTL processes are still required
before the data is suitable to expose to Power BI?
Or, to put it another way, is it not the case that you need to have a clean and properly structured data warehouse
before the data is ready to be massaged and presented by Power BI?
I'd sure value your thoughts and opinions,
Cheers, Donna
Donna KellyDear All,
My original question was:
what's the optimum way of exposing data to the Power BI cloud?
Having spent the last month faffing about with Power BI – and reading about many people’s experiences using it – I think I can offer a few preliminary conclusions.
Before I do that, though, let me summarise a few points:
Melissa said “My initial thoughts: I would expose each dim & fact as a separate OData feed” and went on to say “one of the hardest things . . . is
the data modeling piece . . . I think we should try to expose the data in a way that'll help usability . . . which wouldn't be a wide, flat table ”.
Greg said “data modeling is not a good thing to expose end users to . . . we've had better luck with is building out the data model, and teaching the users
how to combine pre-built elements”
I had commented “. . . end users and data modelling don't mix . . . self-service so
far has been mostly a bust”.
Here at Redwing, we give out a short White Paper on Business Intelligence Reporting. It goes to clients and anyone else who wants one. The heart
of the Paper is the Reporting Pyramid, which states: Business intelligence is all about the creation and delivery of actionable intelligence to the right audience at the right time
For most of the audience, that means Corporate BI: pre-built reports delivered on a schedule.
For most of the remaining audience, that means parameterised, drillable, and sliceable reporting available via the web, running the gamut from the dashboard to the details, available on
demand.
For the relatively few business analysts, that means the ability for business users to create their own semi-customised visual reports when required, to serve
their audiences.
For the very few high-power users, that means the ability to interrogate the data warehouse directly, extract the required data, and construct data mining models, spreadsheets and other
intricate analyses as needed.
On the subject of self-service, the Redwing view says: Although many vendors want tot sell self-service reporting tools to the enterprise, the facts of the matter are these:
v
80%+ of all enterprise reporting requirement is satisfied by corporate BI . . . if it’s done right.
v Very few staff members have the time, skills, or inclination to learn and employ self-service business intelligence in the course of their activities.
I cannot just expose raw data and tell everyone to get on with it. That way lies madness!
I think that clean and well-structured data is a prerequisite for delivering business intelligence.
Assuming that data is properly integrated, historically accurate and non-volatile as well, then I've just described
a data warehouse, which is the physical expression of the dimensional model.
Therefore, exposing the presentation layer of the data warehouse is – in my opinion – the appropriate interface for self-service business intelligence.
Of course, we can choose to expose perspectives as well, which is functionally identical to building and exposing subject data marts.
That way, all calculations, KPIs, definitions, and even field names, and all consistent because they all come from the single source of the truth, and not from spreadmart hell.
So my conclusion is that exposing the presentation layer of the properly modelled data warehouse is – in general - the way to expose data for self-service.
That’s fine for the general case, but what about Power BI? Well, it’s important to distinguish between new capabilities in Excel, and the ones in Office 365.
I think that to all intents and purposes, we’re talking about exposing data through the Data Management Gateway and reading it via Power Query.
The question boils down to what data structures should go down that pipe.
According to
Create a Data Source and Enable OData Feed in Power BI Admin Center, the possibilities are tables and views. I guess I could have repeating data in there, so it could be a flattened structure of the kind Melissa doesn’t like (and neither do I).
I could expose all the dims and all the facts . . . but that would mean essentially re-building the DW in the PowerPivot DM, and that would be just plain stoopid. I mean, not a toy system, but a real one with scores of facts and maybe hundreds of dimensions?
Fact is, I cannot for the life of me see what advantages DMG/PQ
has over just telling corporate users to go directly to the Cube Perspective they want, that has already all the right calcs, KPIs, security, analytics, field names . . . and most importantly, is already modelled correctly!
If I’m a real Power User, then I can use PQ on my desktop to pull mashup data from the world, along with all my on-prem data through my exposed Cube presentation layer, and PowerPivot the
heck out of that to produce all the reporting I’d ever want. It'd be a zillion times faster reading the data directly from the Cube instead of via the DMG, as well (I think Power BI performance sucks, actually).
Of course, your enterprise might not
have a DW, just a heterogeneous mass of dirty unstructured data. If that’s the case,
choosing Power BI data structures is the least of your problems! :-)
Cheers, Donna
Donna Kelly -
Data structure for simulation of message queue
Hello,
I have undertaken a project of simulating the point to point and publish/subscribe protocols of message queueing. This whole project would be done just in Java. There won't be any system level programming. Which data structure in Java would be the most efficient one for the message queue?Hello,
I have undertaken a project of simulating the point to point and publish/subscribe protocols of message queueing. This whole project would be done just in Java. There won't be any system level programming. Which data structure in Java would be the most efficient one for the message queue? -
Data Structure for New General Ledger for mySAP 2004
Guys,
We are preparing to upgrade to mySAP 2004.
mySAP 2004 doesn't have data migration tool for upgrade "classic" ledger to new ledger with advanced functionality. To utilize all new functionality, data should be migrated to the new ledger. Data migration tool in this case should be developed (ABAP). What is the difference in data structure of new ledger?
if anybody who already went thru upgrade and could share some examples of programs for mySAP 2004 upgrade, my e-mail : [email protected]
Thanks in advance,
MikeNow I have the same problem. Anyone has a solution for this matter?
-
Open ended data structure for retrieving SQL data
I have a flex project that reads data from a database and
sends it to my flex app to display in a chart. I need to do this in
Actionscript since I have to dynamically create different charts
based on the database definitions. Up til now, I have inserted the
data for each series into a separate
ArrayList(java)/ArrayCollection(flex) in this way:
BindingUtils.bindProperty(lineSeries, "dataProvider", series,
"pointList");
lineSeries.xField="point1";
lineSeries.yField="point2";
The series variable is a custom Actionscript/Java object
which contains an ArrayCollection called pointList. And the array
is made up of custom DataObjects with two fields, point1 &
point2.
But in the new paradigm, I'd basically like to create a big
array of arrays similar to a database table and then just point the
xField to one column, and the yField to another column. Anyway, it
seems like the problem is that my point1 and point2 in the current
implementation are variable names, but with this new dynamic
structure, I won't be able to create variables on the fly. So how
do I let flex know to look at column 3, for example for the yField?
The attached code is from the Flex documentation and is an example
of what I'd like to emulate (except I need to do it in
actionscript). I'd like to be able to select "month" or "amount",
etc. But coming from the Java side, there is no way for me to name
the array contained in each column.I guess I wasn't clear enough. I am actually serializing a
java object and mapping it to an actionscript object via blazeDS. I
know about how the objects translate, for example for
ArrayCollections in flex, I use ArrayLists in java. The problem is
that I'm not sure how to get the associative aspects of the arrays
in flex into my java data structures in a way that will translate
to the way flex can define arrays, such as Month:"January".
It seems that in flex, you can create an Array and assign it
an element like {month:"January", amount:"450"} and it will create
an Object with variables for month and amount. But I don't know how
to do this on the fly in Java (or in Actionscript for that matter)
when I have no way of knowing beforehand how many variables my
object will need. Each series will need an x and y, though usually
the x will be dates which are the same for each series. So if I had
10 series sharing an axis, I'd need 11 variables in my object-
date, series1, series 2...series 10. -
What is the new data structure for .RTM files in LV8?
We have software that translates our LV
program to another language. Part of that software package reads RTM
files, decodes them, replaces text, and resaves them. That code is
broken for menus that are saved in LV8+, however. What is the new data
structure?
The simple attached zip file has vi's which
are not ours but serve to illustrate the point. If it's run on an
older RTM file, it correctly outputs and XML version of it, on newer
RTM files it just throws an error.
-- This is the second time I posted this. I was curious what the Solution? icon did on somebody elses post and I apparently marked the whole thread as solved (there should really be an undo on that feature btw --
Attachments:
RunTime_Menu_to_XML.zip 33 KBHi Thomas,
What error are you getting when running the program? I was actually able to run it without errors in LV 8.6.
There are ways to programmatically read and customize RTM files in LabVIEW. I hope that the following links will help:
http://zone.ni.com/reference/en-XX/help/371361B-01/lvhowto/customizing_shortcut_menus_programm/
http://digital.ni.com/public.nsf/allkb/17803AA31C8C07C986256CFD0080D609?OpenDocument
Cheers,
Marti C
Applications Engineer
National Instruments
NI Medical -
Storing a lot of data in an indexed data structure for quick access.
I'm designing an app. which will need to store a large amount of data in memory. Records will be flowing into the app. via a socket. The app will receive about 30 records/second which is about 108,000 records/hour and about 600,000 records/day. I need to store the records in an indexed data structure so that I can access them quickly. For example, at 9:00am I will need to access records received at 8:30am, 8:35am, 8:40am, etc. This program will be multithreaded and as I understand Vector is the only data structure that is thread safe. Is Vector my only choice? How do I access objects in a Vector using an index? Is there something better that I can use?
Is Vector my only choice?If you want to access the objects by key then you should use something like a HashMap. But if you want to access them by an array index then an ArrayList would be more appropriate.
as I understand Vector is the only data structure that is thread safeYou can get a thread-safe version of any Collection object by using the Collection.synchronizedCollection method.
How do I access objects in a Vector using an index? I'd suggest you read the API documentation. And probably the Sun tutorial on Collections at http://java.sun.com/docs/books/tutorial/collections/index.html
600,000 records/day. Unless you plan to dump old data after a short period of time, you may want to consider using a database to avoid running out of memory. -
Best data structure for tablemodel
Hi everybody
I'm building an application that has a JTable which gets its data from a resultSet. I'd like to know which data structure is the best for this situation. I've seen some examples that passes the data from the resultset to a Vector or an ArrayList, but if you have a scroollable resultset, why will I instantiate another data structure?
Thanks very much
MauricioHi everybody
I'm building an application that has a JTable which gets its data
from a resultSet. I'd like to know which data structure is the best
for this situation. I've seen some examples that passes the data from
the resultset to a Vector or an ArrayList, but if you have a
scroollable resultset, why will I instantiate another data structure?
Thanks very much
Mauricio I've used TableModels that get their data directly form the ResultSet and had some success. But you are at the mercy of the implementation of the ResultSet. If it is slow your table model will be slow too. I would suggest that you use an array of Object arrays, i.e. Object[][], as your data stucture and copy your result set into it. -
What is the best type of data structure for this type of data?
Animal --> (List of types like Mammal, Reptile, etc)
Mammal --> (List of families)
Etc.
So basically there is a key then a list of other keys with more items.
At first I thought a Map<String, Map<String, Map<String,...etc but the number is unknown so I'm not sure how to structure it.I'd just use a flat Map<String,String> and stick all categories in that
-
Soap2JDBC : Problem while designing data structure for Query
Hi ALL,
I am doing webservices(SOAP) to JDBC(syn) scenario without using BPM.We are sending the request from portal and getting the response from database.
I have lot of Queries.I am going to design based on the database Query.Please find the following Query.
SELECT ALL DBTEST.REQMASTBL.RQTR,
DBTEST.REQMASTBL.RQN_NO,
DBTEST.REQMASTBL.RQN_DATE,
DBTEST.REQMASTBL.APPROVE_BY,
TO_NUMBER(DBTEST.REQMASTBL.RQN_AMT), DBTEST.REQMASTBL.RQN_TYPE,
DBTEST.REQMASTBL.ROS_DATE,
DBTEST.REQMASTBL.RQN_STATUS
FROM DBTEST.REQMASTBL
WHERE DBTEST.REQMASTBL.RQN_STATUS='AP' AND
DBTEST.REQMASTBL.APPROVE_BY LIKE UPPER('GM%') AND DBTEST.REQMASTBL.RQTR='GM211' AND DBTEST.REQMASTBL.RQN_DATE BETWEEN (:frmDate) AND (:toDate) ORDER BY DBTEST.REQMASTBL.RQN_NO
All most all the queries having the same functionality.Could anyone please guide me how to design and how to map the source and the target fields using the Boolean(and,or.like,between) functions.
Now the problem is How to design the data stucture in Integration Repository using this query and how to extract these(and,or.like,between) functions.
urgent response is highly appreciated
rgds,
VeenaHi Gurus
Iam getting the error like this
HTTP error:could not post file
'/XISOAPAdapter/MessageServlet?channel=:BS_SS:soap_communicationchannel&version=3.0&Sender.service=BS_SS&Interface=http%3A%2F%2Fsoap2db1%5Eportalsoap_outbound_messinterface' on server
this is the message iam getting from Portal side.Here iam testing using altova xml spy tool.
in this context
BS_SS is my sender business system
soap_communicationchannel i s my sender communication channel
Sender.service=BS_SS (again my sender business system)
portalsoap_outbound_messinterface is my message interface
I badly required the help from experts.
Regards
Veena -
How To... Change the Data Type for a SELECT-OPTIONS at run time.
Hello,
I am trying to restrict the values available for entry into a SELECT-OPTIONS at run time depending on user input.
The logic is as follows. The user has two input fields. A PARAMETER field which has the type RSDIOBJNM and allows them to choose an InfoObject. And the user has a SELECT-OPTIONS field to allow them to select the Characteristic values for that InfoObject.
I would like the following example to be possible:
The user enters 0MATERIAL into the PARAMETER. When the user clicks on the SELECT-OPTIONS control code will derive a list of possible options the user can enter in the SELECT-OPTIONS. In this case only values found in the master data or at least no values greater than 18 characters.
I have looked at the following function module SELECT_OPTIONS_RESTRICT and this do not appear to be helpful as they only restrict on the signs allowed for the values (unless I misunderstand, it is a complex function module!).
The code I have so far is (thus the user enters a InfoObject into p_char1 and the select options so_char1 should only accept active values of that InfoObject):
declaration of variables for user interface
DATA c_char(32) TYPE c.
declaration of count variable
DATA i_count TYPE i.
declaration of user interface
SELECTION-SCREEN BEGIN OF BLOCK a1 WITH FRAME TITLE text-001.
PARAMETERS: p_ipack TYPE RSLOGDPID.
SELECTION-SCREEN END OF BLOCK a1.
SELECTION-SCREEN BEGIN OF BLOCK b1 WITH FRAME TITLE text-002.
PARAMETERS: p_char1 TYPE RSDIOBJNM.
SELECT-OPTIONS: so_char1 for c_char NO INTERVALS.
PARAMETERS: p_char2 TYPE RSDIOBJNM.
SELECT-OPTIONS: so_char2 for c_char NO INTERVALS.
PARAMETERS: p_char3 TYPE RSDIOBJNM.
SELECT-OPTIONS: so_char3 for c_char NO INTERVALS.
SELECTION-SCREEN END OF BLOCK b1.
Is what I am trying to do possible???
Thanks for any help. Ross.You really want to restrict possible values of a select-option based on another field, not change the length of type of the select-option field, right?
Here is what you do: Code a custom F4 value help for the select-option at event AT SELECTION-SCREEN ON VALUE REQUEST FOR.. The first thing you do here is read the value of the parameter field (p_ipack in your example). You can use function module DYNP_VALUES_READ. Based on this value, you can propose values for the select-option fields. Note that the use can still enter whatever s(he) wishes in to the select-option field without pressing F4. In this case, you will have to code some input validations taking into account the value in the p_ipack field. -
After a fresh install, Firefox loads websites very fast with almost no lag. After a few weeks, it becomes slower and slower. Getting to the point where it wil take aboit 20 seconds to display the Top Sites/Bookmarks window. Then sometimes opening a page, the loading circle stops moving and it freezes the phone. After about 15 seconds everything works again. The only way I found to fix this horrendous problem is to go into control panel for Firefox and Clear Data. I dont like doing that because I lose all of my bookmarks. It seems the Cache always shows "0 bytes" so there never is nothing to clear out. I would like to know where this "Reset" function is for Android. I see they have it for Windows versions..but what about Android?
My phone is an HTC Rezound running stock Android 4.0.3
Thanks in advance.Bruce,<br/>whilst waiting for Roland to get back you may wish to try this
Enter into the address bar ''about:memory'' you get interactive dynamic information about memory usage, and maybe also ''about:compartments'' .
That should at least help you see where the memory is going to. -
Hints how to get a report for "number of active threads over time", e.g.?
Hi there,
We drive a number of WLS setups monitored using Grid Control 11g, and due to the usual economic aspects up to four WLS domains on a single system. To track the development of the load for each single domain we would like to have a report for the active threads, CPU e.g.. In the list of the pre-defined reports we can't find anything matching, so any hints for us based on someone's experience?
TIA
OliverHi Oliver,
I would suggest you to have a look at the below WLST script which would give you the required report of the active threads and it would be send an email too.
Topic: Sending Email Alert for Threads Pool Health Using WLST
http://middlewaremagic.com/weblogic/?p=5433
Topic: Sending Email Alert for Hogger Threads Count Using WLST
http://middlewaremagic.com/weblogic/?p=5423
Also you can use the below script in case of the stuck threads, this script would send you an email with the thread dumps during the issue occurred.
Topic: Sending Email Alert For Stuck Threads With Thread Dumps
http://middlewaremagic.com/weblogic/?p=5582
Regards,
Ravish Mody
Maybe you are looking for
-
The problem is this: In the phase "Create Java Users"
Hello, I am installing Solution Manager 4.0 Release 2 (with Red Hat Enterprise Linux Server release 5.1 and Oracle 10 and my java version is: j2sdk1.4.2_14) The problem is this: In the phase "Create Java Users" I get the next error WARNING 2009-02-13
-
In the Library in iphoto, under Photos, I have 1,896 photos. Sometimes, after working in iphoto, an update will flash through quickly at the top of iphoto. The last one showed 6,131 photos. Where are these photos?
-
Has anyone one else seen problems with there ipad or iphone when you are streaming music from a radio station, were the music just stops and you have to go and start it again. I have this problem on both my ipad and iphone. I have tried different sta
-
IPhone keeps saying it is connected to headphones and it is not
My new Iphone keeps saying the headphones are connected so I can not talk on the phone with it up to my ear. And none of the buttons make any sounds when Iam texting. I tried to reboot it, and restored with my old Iphone on Itunes. Is there someth
-
Basis for creating the dimension for the cube
Dear All, I have one basic query. We can build the ODS and pull data from source system to BW system. But now if i am suppose to build the Cube for my reporting then how do i make decision to build dimension for the cube? I mean what are the paramete