How do you handle and distribute huge amounts of data?

I've got about 315 channels and will be collecting data at about 1300 Hz. Data collection will last about 3-4 minutes. I'm at a loss on how to distribute the data to researchers for evaluation. These researchers will not have LabView or any other NI software to use. I've tried the TDM stuff, and the add-in for excel 2007 but the files are just too large for excel to handle in a timely manner. It took just over three hours to open one of my test files. Anybody have any suggestions on how to manage and distribute files this large?

Frank Rizzo wrote:
Well I'm sitting in Cleveland Ohio right now so I am going to have to disregard your message.....hahhahaha...
I'll be curious to see if Jamal Lewis's yardage improves for you guys this year after having some definite dropoffs with us the last couple.
Though now that he's not playing against you, your defense's numbers against the run should improve.
I'd be surprised if you could get that data to the researchers at all using Excel. It has a limit of 256 columns and 65536 rows. If you had a column per channel and a row per data point, you'd be talking 315 columns and 312,000 rows for 4 minutes of data. I guess you could always break it up into several files being sure to leave some rows and columns to give them a chance to do some data calculation.
Out of curiosity, I created a spread sheet where every cell was filled with a 1. It took a good 30 seconds to save and was over 100 MB. That was probably about 1/10 of the amount of data your dealing with. And going back to my earlier calculations, I would guess that as a text file, that much data would need about 10 bytes per value thus getting you to about 1 GB in text files.
I would ask them what kind of software they will use to analyze these files and what format they would prefer it in. There are really only 2 ways to get it to them, either ASCII text file which could be very large, but would be the most flexible to manipulate. Or a binary file, which would be smaller, but there could be a conflict if they don't interpret the file the same way you write it out.
I haven't used a TDM file add-in for Excel before. So I don't know how powerful it is, or if there is a lot of overhead involved that would make it take 3 hours. What if you create multiple TDM files? Let's say you break it down by bunches of channels and only 20-30 seconds of data at a time. Something that would keep the size of the data array within the row and column limits of Excel. (I am guessing about 10 files). Would each file go into Excel so much faster with that add-in that even if you need to do it 10 times, it would still be far quicker than one large file? I am wondering if the add-on is spending a lot of time figuring out how to break down the large dataset on its own.
The only other question. Have you tried the TDMS files as opposed to the TDM files? I know they are an upgrade and are supposed to work better with streaming data. I wonder if they have any improvements for larger datasets.

Similar Messages

Handling Huge Amount of data in Browser

Some information regarding large data handling in Web Browser. Browser data will be downloaded to the
cache of local machine. So when the browser needs data to be downloaded in terms of MBs and GBs, how can we
handle that ?
requirement is as mentioned below.
A performance monitoring application is collecting performance data of a system every 30 seconds.
The size of the data collected can be around 10 KB for each interval and it is logged to a database. If this application
runs for one day, the statistical data size will be around 30 MB (28.8 MB) . If it runs for one week, the data size will be
210 MB. There is no limitation on the number of days from the software perspective.
User needs to see this statistical data in the browser. We are not sure if this huge amount of data transfer to the
browser in one instance is feasible. The user should be able to get the overall picture of the logged data for a
particular period and if needed, should be able to drill down step by step to lesser ranges.
For e.g, if the user queries for data between the dates 10'th Nov to 20'th Nov, the user expects to get an overall idea of
the 11 days data. Note that it is not possible to show each 30 second data when showing 11 days data. So some logic
has to be applied to present the 11 days data in a reasonably acceptable form. Then the user can go and select a
particular date in the graph and the data for that day alone should be shown with a better granularity than the overall
graph.
Note: The applet may not be a signed applet.

How do you download gigabytes of data to a browser? The answer is simple. You don't. A data analysis package like the one you describe should run on the server and send the requested summary views to the browser.

How can we transfer huge amount of data from database server to xml format

hi guru
how can we transfer huge amount of data from database server to xml format.
regards
subhasis.

Create ABAP coding
At first we create the internal table TYPES and DATA definition, we want to fill with the XML data. I have declared the table "it_airplus" like the structure from XML file definition for a better overview, because it is a long XML Definition (see the XSD file in the sample ZIP container by airplus.com)
*the declaration
TYPES: BEGIN OF t_sum_vat_sum,
          a_rate(5),
          net_value(15),
          vat_value(15),
         END OF t_sum_vat_sum.
TYPES: BEGIN OF t_sum_total_sale,
        a_currency(3),
        net_total(15),
        vat_total(15),
        vat_sum TYPE REF TO t_sum_vat_sum,
       END OF t_sum_total_sale.
TYPES: BEGIN OF t_sum_total_bill,
        net_total(15),
        vat_total(15),
        vat_sum TYPE t_sum_vat_sum,
        add_ins_val(15),
        total_bill_amount(15),
       END OF t_sum_total_bill.TYPES: BEGIN OF t_ap_summary,
        a_num_inv_det(5),
        total_sale_values TYPE t_sum_total_sale,
        total_bill_values TYPE t_sum_total_bill,
       END OF t_ap_summary.TYPES: BEGIN OF t_ap,
        head    TYPE t_ap_head,
        details TYPE t_ap_details,
        summary TYPE t_ap_summary,
       END OF t_ap.DATA: it_airplus TYPE STANDARD TABLE OF t_ap
*call the transformation
CALL TRANSFORMATION ZFI_AIRPLUS
     SOURCE xml l_xml_x1
     RESULT xml_output = it_airplus
     .see the complete report: Read data from XML file via XSLT program
Create XSLT program
There are two options to create a XSLT program:
Tcode: SE80 -> create/choose packet -> right click on it | Create -> Others -> XSL Transformation
Tcode: XSLT_TOOL
For a quick overview you can watch at the SXSLTDEMO* programs.
In this example we already use the three XSLT options explained later.
As you can see we define a XSL and ASX (ABAP) tags to handle the ABAP and XML variables/tags. After "

How do you handle multiple at the same address?

My wife just got an iMac. She is using the Birthday feature in Address Book and iCal to track our friends and family birthdays and anniversaries.
Question:
How do you track it when multiple people are at the same address? For instance, our friends are a family of four, but they all live at the same address. Do you need to make a separate Address Book card for each of them, even the 1 year old baby? Or can you track multiple people in a single card with custom fields?
If you do need to make multiple cards, then how do you handle mailing lists? For instance, if I make separate cards for all four of them, how do I make sure I only send on Christmas card when I use my Mac to print labels?
Thanks for any advice.

This is really an issue that you're going to have to resolve yourself since Apple's address book isn't really built with this in mind - darn it. For me there are three scenarios. First, a singleton. No problem. Second a family for which all members share the same information - acquantences to whom I don't send birthday greetings. These get one card and I'll put in both (or all) their names in the first name field (as in Bob and Jean). If there are children, in the last name field I'll add 'and Ken' or 'and the kids'. Third, a family for which some fields are different - perhaps cell phone number, birthday, etc.
For mailing purposes, I create a holiday group and put the people I'll be sending cards to into the group.

How do you handle an incoming call while playing a game?

If you're playing a game on your iPhone, how do you handle an incoming call? A call will stop my app from running, but how do I save the state and restore it when the user starts the app again?
I have a bunch of objects on-screen and I need to record where they are so that the user doesn't lose the progress of their game.
I've looked through the SDK, but I can't see anything obvious. Any ideas?

I don't have a device to test with, but I suspect you'd want to save state in one of the UIApplicationDelegate callbacks:
applicationWillResignActive:
applicationWillTerminate:

Re: How do you create and use "common" type classes?

Hi,
You have 2 potential solutions in your case :
1- Sub-class TextNullable class of Framework and add your methods in the
sub-class.
This is the way Domain class work. Only Nullable classes are sub-classable.
This is usefull for Data Dictionary.
The code will be located in any partition that uses or references the supplier
plan.
2- Put your add on code on a specific class and instanciate it in your user
classes (client or server).
You could also use interface for a better conception if needed. The code will
also be in any partition that uses or references the supplier plan where your
add on class is located.
If you don't want that code to be on each partition, you could use libraries :
configure as library the utility plan where is your add-on class.
You can find an example of the second case (using a QuickSort class,
GenericArray add-on) with the "QuickSort & List" sample on my personal site
http://perso.club-internet.fr/dnguyen/
Hope this helps,
Daniel Nguyen
Freelance Forte Consultant
http://perso.club-internet.fr/dnguyen/
Robinson, Richard a écrit:
I'm relatively new to forte and I'd like to know how can you handle utility
type classes that you want to use through out your application? Ideally
what I want is a static class with static methods.
Let's say that I have a StringUtil class that has a bunch of methods for
manipulating strings.
My problem is that we have code that runs on the client and code that runs
on the server. Both areas could use the StringUtil class, but from what I
understand, I have to create StringUtil in a plan and then create a server
object of type StringUtil. The server object will eventually get assigned
to a partition. That's not good since I really want the server object to
physically reside at the server end and at the client end. (Actually, I
don't want a server object, I just want to invoke a static method of a
static class).
Any clues on how to solve this problem would be appreciated.
Also, what is the url at Sage-it that has a summary of all emails that have
been posted to [email protected]? Perhaps this question has been
answered previously.
Thanks in advance
To unsubscribe, email '[email protected]' with
'unsubscribe forte-users' as the body of the message.
Searchable thread archive <URL:http://pinehurst.sageit.com/listarchive/>-
To unsubscribe, email '[email protected]' with
'unsubscribe forte-users' as the body of the message.
Searchable thread archive <URL:http://pinehurst.sageit.com/listarchive/>

Hi Richard,
Your question about "utility classes" brings up a number of issues, all of
which are important to long-term success with Forte.
There is no such thing as a static method (method that is associated with a
class but without an implicit object reference - this/self/me "pointer") in
TOOL, nor is there such thing as a global method (method not associated
with a class at all). This is in contrast to C++, which has both, and
Java, which has static methods, but not global classes. Frequently, Forte
developers will write code like this:
result, num : double;
// get initial value for num....
tmpDoubleData : DoubleData = new;
tmpDoubleData.DoubleValue = num;
result = tmpDoubleData.Sqrt().DoubleValue;
tmpDoubleData = NIL; // send a hint to the garbage collector
in places where a C++ programmer would write:
double result, num;
// get initial value for num....
result = Math::Sqrt(num);
or a Java programmer would write:
double result, num;
// get initial value for num....
result = Math.sqrt(num);
The result of this is that you end up allocating an extra object now and
then. In practice, this is not a big deal memory-wise. If you have a
server that is getting a lot of hits, or if you are doing some intense
processing, then you could pre-allocate and reuse the data object. Note
that optimization has its own issues, so you should start by allocating
only when you need the object.
If you are looking for a StringUtil class, then you will want to use an
instance of TextData or TextNullable. If you are looking to add methods,
you could subclass from TextNullable, and add methods. Note that you will
still have to instantiate an object and call methods on that object.
The next issue you raise is where the object resides. As long as you do
not have an anchored object, you will always have a copy of an object on a
partition. If you do not pass the object in a call to another partition,
the object never leaves. If you pass the object to another partition, then
the other partition will have its own copy of the object. This means that
the client and the server will have their own copies, which is the effect
you are looking for.
Some developers new to Forte will try to get around the lack of global
methods in TOOL by creating a user-visible service object and then calling
methods on it. If you have a general utility, like string handling, this
is a bad idea, since a service object can reside only on a single
partition.
Summary:
* You may find everything you want in TextData.
* Unless you anchor the object, the instance will reside where you
intuitively expect it.
* To patch over the lack of static methods in TOOL, simply allocate an
instance when required.
Feel free to email me if you have more questions on this.
At the bottom of each message that goes through the mailing list server,
the address for the list archive is printed:
http://pinehurst.sageit.com/listarchive/.
Good Luck,
CSB
-----Original Message-----
From: Robinson, Richard
Sent: Tuesday, March 02, 1999 5:44 PM
To: '[email protected]'
Subject: How do you create and use "common" type classes?
I'm relatively new to forte and I'd like to know how can you handle utility
type classes that you want to use through out your application? Ideally
what I want is a static class with static methods.
Let's say that I have a StringUtil class that has a bunch of methods for
manipulating strings.
My problem is that we have code that runs on the client and code that runs
on the server. Both areas could use the StringUtil class, but from what I
understand, I have to create StringUtil in a plan and then create a server
object of type StringUtil. The server object will eventually get assigned
to a partition. That's not good since I really want the server object to
physically reside at the server end and at the client end. (Actually, I
don't want a server object, I just want to invoke a static method of a
static class).
Any clues on how to solve this problem would be appreciated.
Also, what is the url at Sage-it that has a summary of all emails that have
been posted to [email protected]? Perhaps this question has been
answered previously.
Thanks in advance

SMTP traffic blocked by ISP how do you handle it ?

I have recently installed the OCFO 10.1.3.07. We were using POP account previously and i had 2 account created in outlook (one using my SMTP server as outgoing mail server and the 2nd using the user ISP SMTP server) to let users send emails from home by letting them select the account in outlook before sending their mail without having them to configure anything.
I am now stuck here. I tried replicating the same kind of setup using OCFO and a secondary IMAP4 account but it doesnt work. The mail stays in the outbox. The only way to send from their homes is to run the configuration wizard and change the SMTP server adress.
How do you handle this? am i taking the wrong approach here?
Thanks for any inputs.

For anyone who might be interested,
I have submited a SR to oracle support and the workaround to this issue is to Create a 2nd mail profile and configure the OCFO with a different SMTP server within that profile.

Interview Q: 'how did you handle delta'

Hi BW experts,
I've been asked this question in the interviews, 'how did you handle delta?' I know how to create delta. What is the right answer for this type of question? should I start explaining what/how we did in my previous company?
I'd appreciate and assign points, if someone can provide the answer for this.
Thanks,
Sam

The answer depends on what you have been talking about, I mean the previous discussion you had with the the interviewer. There are different way of handling delta... example:
1. The extractor itself handles delta and it send you the before and after image of the changes.
2. sometimes you load the data direclty to ODS and depending on the key you have in ods, the delta is being handled.
3. In somecases, you may not need to worry about handling delta at all (full load)... going back to the quesion you asked, that depends on the discussion you had with him/her.
thanks.
Wond

How do you handle change in user names if folks get married or divorced

If an application is using the apex authentication scheme, how do you handle change in user names if folks get married or divorced. The reason I ask the question is that all of apex uses v('app_user'), which is the user_name. If I have transactions done by this user and then their name changes, which means so does their id, how do I get all of their transactions, before and after the name change. Traditionally I would expect to use user_id, not sure if this feature is still available. I can always create a custom auth scheme, but that is a lot of additional work. Any ideas?
Thanks,

Appreciate your help and time Jari, below is my game plan, unless someone has a better idea,
I plan on doing the following to use user_id for auditing of transaction records:
1. Get the user_id to use in the tables audit trail fields
user_id := apex_util.get_user_id(v('app_user'));
OR
user_id := apex_util.get_current_user_id;
2. If a user changes their name then use
apex_util.set_last_name(p_user_id, p_new_lastname);
apex_util.set_username(p_userid, p_new_username);
Thanks,

How do you select and move more than one bookmark at a time? Shift+Click does not select multiple items that are next to one another in a list because the item

How do you select and move more than one bookmark at a time?
Shift+Click does not select multiple items that are next to one another in a list because the items open in firefox before this happens.

Use the bookmarks library. You may use Shift +Click, and Ctrl + Click to create groupings of selected bookmarks to drag and drop.
* one method of opening the bookmarks library is keyboard shortcut <br /> Ctrl+Shift+B (Windows)
*see also [[How to use bookmarks to save and organize your favorite websites]]
*and [[Use bookmark folders to organize your bookmarks]]

How do you add and remove links to websites in folders on Safari?

How do you add and remove links to websites in folders on Safari?

For clarification, works like a bookmark of website only it is listed in toolbar with "Most Visited", "Latest Headlines", "News", and "Popular". I have done it before but can't find it for the life of me.
Thanks!

I have huge amount of data on a windows external drive and want to transfer to a Mac drive. Does anyone know an easy way to do this? I have almost 2TB of data to transfer. Thanks.

I have huge amount of data 2TB on a windows Fantom external drive and want to transfer to a Mac drive. Does anyone know an easy way to do this? Thanks. I have an IMac 3.5 GHz Intel Core i7. I haven't bought a Mac external yet.

Move your data to a new Mac - Apple Support

How do you Design and Develop?

How do you Design and Develop?
I know this varies based on alot of things such as knowledge, preference, purpose, resources, etc..but I am curious how different (level of expertise) designers actually prefer or implement their websites. I decided to do this post after two different forums discussions regarding 1.Templates and PHP and 2. Backend Development.
1. How do you generally approach a project?--Wireframe, Freehand (as it comes), PSD template, etc.--Do you have a specific 'methodology' or procedure?
2. What are 2-3 of your main resources / sites / applications etc that you use (and for what?)
Specifically: What do you use (and how) to get up a website? (I know again this depends on a variety of things, but in general..) *also if you would like to comment on your expertise or level maybe helpful.
Do you prefer (Pros/ Cons of:) What advice would you give regarding below?
Your own HTML/CSS/Script based site that you mostly (if not completley) built. (More control and original, but harder / tedious for beginner)
HTML Temlplate (tweak the HTML and CSS)--non framework type (Quick and easy, but usually poor quality, outdated?)--but some can servde as good starting point.
Wordpress (And others like Drupal, Joomla) (Somewhat quick and easy access to tweak a quality WP Theme with ability for alot of features--forms, search, databases, etc (otherwise might have trouble with advanced features w/o good working knowledge og PHP and MYSQL, maybe others?)--but kind of a 'workaround' for real HTML design?
Frameworks types like Foundation, Bootstrap, Skeleton
A combination or hybrid of methods? -- or another method I have overlooked?
Thanks very much for any insight you can offer, hopeully this will lead to a vibrant discussion.-

The websites I build are 95% PHP/MySQL. They are heavy on business logic. I sometimes work with designers, SEO "experts" and (rarely) other coders. I almost never work on the content of websites. The clients do that themselves through the CMS I build for them.
I write every line of code myself except for the use of jQuery and a few PHP code classes adapted from others. I can build a complex website in a matter of minutes by reusing the code libraries and patterns I have developed from previous projects.
I don't use a PHP framework because I have developed my own "pseudo-framework" for the particular kinds of sites I build, but I respect the frameworks that are out there. They can teach you excellent coding habits.
In starting the design and logic of a site, the most essential question is: What is the website (or website project) supposed to accomplish? My clients tell me what they want to accomplish, but I usually decide how to get there. This holds for both business logic and design, because design also must serve the business objectives.
My website projects generally take four months to two years to complete. The client usually dedicates one employee to work as a liaison between me and the rest of the company. Because the sites I build are heavy on the business logic, a great deal of time is spent on testing and debugging. I make testing the responsibility of the client so I can't be held liable if, for instance, a terrible accounting mistake is discovered in the code logic.
I like to use Adobe Illustrator for design development because I became an Adobe Illustrator ACE in 2000; however, I can see that Fireworks is probably the better tool. Photoshop is an important secondary tool.
For general site management and code writing I use both Dreamweaver and ZEND Studio; however, I use Dreamweaver less and less.
I now require all my clients to have their websites (and also domain name registrations, since my company is a domain name reseller) hosted on my VPS server. This prevents anyone from stealing code I have worked for years to develop and also establishes a commitment between myself and the client. When you are starting a web development business, you can't demand this kind of commitment from the client. Even a lot of great seasoned developers can't. It depends on your reputation within the specific industry(ies) you service.

If you've changed your startup options on mac to start in windows, how do you fix and go back to mac on startup?

if you've changed your startup options on mac to start in windows, how do you fix and go back to mac. I can't get my mac to switch back to mac, it is like stuck in widows mode, I have tried holding down the options key upon start up and that did not fix it either.
Please help! Thank you so much!

The Apple drivers you install in Windows also puts a Startup Drive icon in the Task Bar. Choose that to change the startup disk back OS X.

How do you drag and drop using mac laptop?

how do you drag and drop using mac laptop?

As said above, you can click down, ons use the other finger to drag the object. I had turned on the setting to use three-finger drag, so you can drag objects with three fingers. To change gesture settings, go to the system preferences>trackpad. Here you can also see all of the gestures of Mac OS X.

How do you handle and distribute huge amounts of data?

Similar Messages

Maybe you are looking for