Best log file format for multivariable non-continous time series

Databases or TDM(S) files are great, but what if you cannot use a database (due to the type of target) and TDM files are (seems) unsuitable because the data does not come in blocks of continous time series. What is the best file option for data logging?
Scenario:
The number of variables you are going to log to a file can change during run-time
The data is not sampled at fixed intervals (they have been deadband filtered e.g.)
The files must be compact and fast to search through (i.e. binary files with known positions of the time stamps, channel descriptions etc.)
Must be supported on compact fieldpoint and RIO controllers
Right now we use our own custom format for this, but it does not support item no. 1 in the list above (at least not within the same file) and it would be much nicer to have an open format that other software can read as well.
Any suggestions?
MTO

I did some tests of the performance. For a months worth of data (2592000 rows) with 4 channels, I got the following results when reading all of the data:
1. TDMS file written as blocks of 60 values (1 minute buffers):1,5 seconds.
2. As test 1, but with a defrag run on the final file: 0,9 seconds
3. As test 1 & 2, but with all the data written in one operation: 0,51 seconds
3. Same data stored in binary file (1 header+2D array): 0,17 seconds.
So even if I could write everything in 1 go (which I cannot), reading a month of data is 3 times faster with a binary file. The application I have might get a lot of read-requests and will need to read much more than 1 month of data - so the difference is significant (reading a year of data if stored as monthly files would take me 12-18 seconds with TDMS files, but just 2 seconds with a binary file.
Because I'll be writing different groups of data at different rates, using the advanced api to just get one (set) og header(s) is not an option.
TDMS files are very versatile, it is great to be able to dump a new group/channel into the file at any time, and to have a file format that is supported by other applications as well. However, if the number of writes are many and the size of each write is (has to be) small the performance gets a serious hit. In this particular case performance trumphs ease of use so I'll probably need to rewrite our custom binary format to preallocate chunks for each group (feature request for TDMS? :-) ).
MTO

Similar Messages

Best graphic file formats for use in FC

The FC User's Manual indicates that FC can use PICT, TIFF, TGA, PDF, Photoshop (I assume .psd) or JPEG files for images.
What is the best file format, most compatible with FC and DV? I understand that image files ought to be 720 x 480 px for standard video.
What is the preferable format and why?
I made a test set of cels for animating in FC. I made them in Photoshop at 720x480 px and saved them in PICT format in a folder with sequential filenames such as pic0001.pct, pic0002.pct, pic0003.pct, etc. They worked fine.
Would jpgs be preferable to picts for this?
When using jpgs what "quality" setting is best?
When using .psds, ought they to be "flattened" to one layer before use in FC?
Thanks,
George

I would prefer 'non-compressed' formats as pict or tiff ... if you have the choice.
although 'perfect' on screen, jpegs, after conversion into any video-codec, you'll soon notice 'banding'...
and 'saving disk space' should be no issue while movie making.
for sure, it makes no sense, to convert a jpg into tiff before import
.psd should only be 'flattened' if you don't need the layers as tracks ... example: I created a title in PS with a shadow on a separate layer. in FC/e I animated the title with a super-fast 'moving from outside into frame' ... and the shadow comes 'delayed' => that fast! ... old Looney Toons effect.

Line Drawing best file format for import in InDesign

i was wondering what is the accepted and recommended file format for files to be imported in InDesign. I was under the impression that AI files should be saved as EPS but I just read an article that says that Adobe's Dov Isaacs said that eps will be shortly obsolete.What should I do. I work on curriculum that contains both photographs and line drawing and we use tiff for the former and eps for the latter. Most of our line drawing is done in AI. Howeverwe have a cartoonist that sends us files in jpg format. What would be the best file format for him and for us as we draw in AI?
Article link: http://www.prepressure.com/library/file-formats/eps
The future of the EPS file format
EPS is rapidly becoming an outdated file format which is being replaced by PDF just like PostScript itself is also being phased out and replaced by PDF. Don’t just take my word on this. Here is what Dov Isaacs from Adobe said in a discussion on a PrintPlanet forum about the future of PostScript: “ …Adobe will continue to support EPS as a legacy graphics format for import of non-color managed, opaque graphical data into Adobe applications (such as InDesign and Illustrator). Although we certain do not recommend that new graphical content be stored in EPS format (except to satisfy the need to import data into page layout programs that aren’t quite PDF-centric — no need to mention names here!), our user base should feel comfortable that there is no need to worry about a need to convert their very sizable libraries of EPS-based graphic assets.”

Yes, I drag the AI file in.
If after it's in place I need to change I can just right click and chose edit original make the changes and save and the illustration in inDesign will reflect the changes.
On the other hand I can make changes to the original and all iterations of the file will automatically update, say if I used the illustration in several documents.
As an example I had completed a set of documents for a new company, i.e., biz cards, brochures, product labeling, stationery, etc. and in their wisdom they decided to make changes in their logo, which, of course, is on everything. I opened the AI file with the logo made the changes saved and all the documents then reflected the changes. The only caveat to do this seamlessly is to make sure that your file logo size is the same as the original or you might have to tweak placement in the other documents
It is this interoperability that to me makes the Creative Suite so much more than the sum of their parts.
Craig

Best Hi Def File Format for PE 10 on a PC?

I am running Premiere Elements 10 on a PC, Windows 7 Professional 64 bit, two internal 1 TB Hard Drives, 16 gigs of RAM, with Blu Ray Burner drive. I have done a fair amount of video editing using PE 10 on this rig with standard def material, but now I hope to do some high def work. I plan to have some old family 16mm home movies transferred to hi-def media and do some minor editing (color / contrast correction and maybe some simple transition effects) and then burn the edited files to Blu ray disc.
I am all ready to send my 16mm footage off the the transfer house, but I need to know what would be the best file format for my set up. The formats they have available are DPX Image Sequence, TIFF Image Sequence, and Quicktime (either 422 HQ or 4444 HQ). I am assuming the preferred format would be either of the Quicktime files, and I have downloaded short (about 10 to 15 second) sample files in both 422 HQ and 4444 HQ and was able to import each into PE 10 with no problem and successfully do various edits on each.
But I have read a good number of posts on this and other forums from PC owners who are running Premiere Elements and have had problems either loading Quicktime 422 & 4444, or editing them once they were loaded into PE.
Can anyone comment on this for me? Basically, all I want to know is what Hi Def file format(s) I can reliably use on my computer with PE 10. I don't want to pay several hundred dollars (or possibly more) for a digital transfer from my 16mm footage that turns out to be useless to me.
Rob McKay

ATR -
Thanks for your very informative reply. I'm afraid the transfer house doesn't offer samples.
Since you think the ProRes 422 would be best for me, here is what I have done in my own tests:
1) Downloaded short ProRes 422 HQ clips into files on my computer hard drive from several websites. The longest of these clips is 30 seconds.
2) Using the "get media" tool in PE 10, I've brought these clips into PE 10.
3) Dragged the clips into the project timeline. At first I got a popup message saying (and I have paraphrased this) the "file format is not supported by PE 10. Would you like the program to convert it to a format supported by PE 10?" I clicked "yes" and the clip appeared in the timeline. Later on I stopped getting this popup message. I've no clue what format the program might be converting the ProRes 422 HQ files to.
4) After I had a couple minutes of short clips in the timeline, I copied and pasted some of them over and over in the timeline until my entire video time was about 30 minutes in length. Reason for this was simply because I am trying to replicate as best I can what my file from the Transfer House will be like - not 2 minutes in length, but closer to 30 minutes.
5) Did a few minor editing effects on the timeline of the sort I would be doing on my Transfer House file - contrast and color correction, time stretch /speed change, etc.
6) Went to "Share" tab, clicked on "Disc - Burn to Blu Ray". In the Presets dropdown menu, I chose "H.264 1920x1080i NTSC Dolby" (Note - Only four choices appear. The one I chose, as well as "H.264 1440x1080i NTSC Dolby" and the same two again except in PAL Dolby.)
7) Burned a BD-R of the project in Blu ray burner.
I was very pleased with the result - image was nice sharp quality, smooth moving (no dropped frames) and filled my 4 foot x 8 foot screen when played on my Blu Ray player through an Epson projector.
Rob

Best File Format For Online Line Art?

I thought the conventional wisdom was that all line art for the screen should be saved as GIFs, all photographic line art should be saved as JPEGs.
Then PNGs came along and offered an alternative to GIFs.
I've been exploring this because I've been having a problem when exporting line art from Illustrator 8 as a PNG and opening it in Fireworks MX, where it appears washed out.
I did a bit of research among comic strip artists -- the art I'm exporting from Illustrator is a comic strip -- and came up with conflicting results.
Dilbert seems to save black and white line art as GIFs, e.g. http://dilbert.com/comics/dilbert/archive/images/dilbert2008611230313.gif
... But its Sunday colored strips seem to be saved as JPEGs, e.g. http://dilbert.com/comics/dilbert/archive/images/dilbert2008040261966.jpg
Meanwhile Donnesbury seems to save colored comic strips as GIFs, e.g. http://images.ucomics.com/comics/db/2008/db080406.gif
Any thoughts, suggestions?
Thanks.

I assume your line art is drawn as vector art in Illustrator, correct?
As far as I'm concened, the "best file format for online line art" would be .swf, because it would retain its scaleable vector nature, and the Flash Player is most likely embedded in any reasonably recent computer's browser.
But a little simple experimentation would be in order regarding file size. SWF (as exported from Flash, not necessarily AI) is very efficient. But the complexity of the line art could still make a given image require more bandwidth as .swf than as a rasterized version.
The former conventional-wisdom regarding PNG is the old "not supported in older browsers" saw. I see even Firworks's online help still says:
> However, not all web browsers can view PNG graphics.
Personally, I think its way past time to stop cow-towing to outdated browsers and use PNGs anyway. So if it were me, and the nature of my line art required rasterization for online use, I would use PNG for its obvious advantages. But then, my livlihood is not affected by some minority of outdated web viewers not being able to view my PNGs.
I would think that whatever color problems you are having in getting your AI files exported to PNG can be worked out with a little investigation/experimentation. (PNG supports color management profiles, so that alone may be your problem, and may be easily fixed in your workflow.) GIF is by definition limited to a subset of colors. JPEG is by definition a lossy compression format. So there are certainly precise color-accuracy issues with those overused formats, too.
But again, that's just me. What little I do in web graphics that would involve this issue doesn't really affect my livelihood. Lowest-common-denominator compatibility may be more important to you.
> Gilbert seems to...
> Doonsbury seems to...
The two images are very close to the same size (132K & 125K). They both include contone grads. Between JPEG and GIF, I would expect JPEG to more consistently yield acceptable results. Since GIF is limited to an 8-bit lookup table, I would expect aberations such as unwanted banding to be more likely in GIF if the artwork contains many colors and alot of highly-rendered shading.
Since you specify
line art, though, that would rule out contone fills, and you could use GIF with a very small color table.
On the other hand, you do not specify whether you want to use transparency in your line art images. If you do, PNG would be far superior, because it supports true alpha transparency. In a GIF, if you can only make one particular color transparent. So even with line art, if the line art is antialiased at all, you'll have the ugly halos so commonly seen in GIFs when displayed on a wrong-colored background.
The only functional "advantage" that I see in GIF over PNG is its multi-image capability, for quick & dirty frame-by-frame animation. My personal animation efforts focus entirely on Flash/SWF, and use scripted animation whenever possible to avoid frame-by-frame. But I recognize the simplicity convenience of animated GIFs for web banners and those (often-annoying) moving icons and such.
JET

For iPhone and Android, what is the best file format for watching videos?

For iPhone and Android, what is the best file format for watching videos?

You can try these setting. They are from CS5.5 but should work in CS4.
Make sure you check Use Max. Render Quality.

Best or better file formats for editing in PPRO CS4

I'm currently working with MPEG (from tape transfer) and M2T ( recorded with Adobe Onlocation) files in CS4.
I switch between Mac and PC depending on whcih system is available. (both: 8 gigs ram , dual quads 3GHZ, graphics ATI 4850 &/or quadra FX)
Regardless of the OS or system configuration I'm still experianceing choppy video in the Program monitor during editing.
So I guess I'm wondering if I should be useing a differant file formats for editing.
end use is HD web video that is served via Delve networks (720P).
Detailed, positive ideas and commentary appreciated.
thanks JamesNYC

If you're capturing via OnLo and it's coming up as .m2t then OnLo sees the device as an HDV device. This is a temporal codec and as such, decoding can be difficult if your system isn't up to the task. Since your CPU and memory are up to the task, I would ask about your drive subsystem and exactly which OS you are using? If you're using a 32bit flavor of Windows, you're not able to use the extra memory and I would suggest you upgrade to Win7 64.
hope this helps,
Dennis

What is THE best video file format to import into Premiere Pro CS6?

What is THE best video file format to import into Premiere Pro CS6? I am tired of the guess work of "this might work if you do this." Just tell me which is, flat out, the easiest for Premiere to read?? I generally animate in Adobe Flash, and some times with a Live camera. Premiere seems to pick and choose which video file it wants. I import from the File drop down box in the upper left hand corner, then move the video file to timeline. Under the "scrubber" there is a thin red, (or when rendered) a green line. The videos I put in the timeline show a thin yellow line which means Premiere is confused. So, in basic terms, which video format is MOST commonly accepted with the least amount of trouble/hassle?

Premiere has a function now where you can import a clip and drag onto a timeline and it will prompt you with a message about changing the timeline settings to match your clip, (or if there is no timeline it will automatically create one that matches your clip.) This is in CC of course so you might consider upgrading. They have fixed a lot of compatibility issues from CS6 to now. If you don't want to get all of CC for 60/month then you can at least shell out 20/month on premiere. It's worth it. They've made it where you don't have to worry about 99% of all video formats. They just work.

Oracle Retail standard file format for batches - specification needed

Is there any accurate specification of Oracle Retail standard file format for batches, such as EdiUpcat.pc (possibly with examples)?
I am trying to produce a file with supplier cost changes for Cost Change Batch (ediupcat.pc).
In "Operations Guide - Batch Overviews and Designs - Volume 1" I found some brief description, though it is not clear to me
1) what should I do with empty values (obviously, pad with appropriate number of spaces?)
2) What if some string value has less characters than specified? Should I left-pad or right-pad?
3) What does Number(12,4) means? How many characters exactly? 17? Why then values in some fields are multiplied by 10000 (so that to obtain integer value) while in others they go in Number(12,4) notation?

Hi Roman,
I am glad to be of help.
Try to see it like this:
A number 20 is of char-length 20 in a flatfile, and the value is what it is 1:1.
A number 12,4 is of char-length 12 in a flatfile, but needs to be divided by 10^4 (10000) to get its real value.
Indeed there is a funny asymmetry in the documentation how the decimals are handled - but we needed to read carefully already anyhow... :)
I always like to construct a file by hand in an editor (vi, whatever) and then run it through the program, and check the errors in the error file.
Good luck and best regards,
Erik
Edited by: ErikYkema on Oct 13, 2011 3:01 PM

Firefox for Mac preferences shows .qif file format for quicken as "quicktime image format"-wrong application and won't let me select anything else

Post to Firefox forum; Oct.2011
Re: Quicken 2007; Mac OS 10.5
Problem: can’t import or download Qif format files into quicken from bofa or Chase. Mac can’t recognize; how do I set preferences in Firefox (or Safari).
https://support.mozilla.com/en-US/questions/new?product=desktop&category=d1&search=Firefox+for+Mac+preferences+shows+.qif+file+format+for+quicken+as+%22quicktime+image+format%22-wrong+application+and+won%27t+let+me+select+anything+else&showform=1
Hello,
I previously used Firefox on my old Mac G4 with OS 10.3.9; the above issue existed with that system and Firefox 2.00.2 (or similar). I then "tried" to move to a G5 Mac with OS 10.5, hoping that an updated Mac Operating system and updating Firefox would solve that problem.
I'm now working on my son's MacBook Pro, OS 10.5.8, 2.4 ghz Intel Core 2 Duo; 2gb Memory; 2 gb 667 Mhz DDR2 SDRAM.
I still have a problem in Firefox Preferences trying to download .qif files. BofA Mortgage accounts only allow downloading of .qif files. My .qfx download works fine, but the error message is as follows:
The error message is that this is a corrupted file or one that Preview doesn't recognize.
and it is still a problem when trying to download Quicken .qif format files for our B of A mortgage.
The preferences for Firefox under File Helpers won't let me name the extension I want; instead, it CONFUSES .qif (Quicken Interchange Format) with .qtif (Quicktime Image Format) for Mac.
Therefore, when I download .qif files and try to import them to Quicken, I get a message that this is a corrupted file:
thanks for any help.
Val in Seattle

Well, in the end, I just gave up and deleted the photos, in the hope that it was something to do with them. Took some more photos and they seem to have imported without any problems at all.
Given that the same irksome photos loaded on the wife's macbook without so much as a murmur, how weird is that?
Still- all's well that ends well. Thanks for the suggestions- much appreciated.
Matt

How do I change the default file format for saving files in Preview

I often take screen clips and save them as files to my desktop, then upload them to my website or include them in emails. The default file format for Preview seems to be TIFF, but I'd prefer jpg. How do I convince Preview to use jpg as the default file format?
TIA,
Thad

I should have been a little clearer...
If I'm emailing a screen-clip, I can just paste it right into the email, but I post on a lot of discussion boards and it's often nice to be able to include a screen shot in those. My preferred way to do that is to post the clip at my personal website and then put a link in the post. It gives me the ability to delete the image whenever I want (a plus for any security freak) and it works on every discussion board I know. Some boards allow attachments, some don't. Plus I don't want to have to remember different d/b file attachment procedures. Anyway, when I make a screen-clip, I need to temporarily save it (usually to my desktop), then ftp it to my website. I use Preview to do that. It's simple. But I'd like it to default to .jpg instead of .png or .tiff. (FWIW, I just tried it and the default was .png, but I swear I did something similar earlier that defaulted to .tiff when I first tried to save the newly-created file).

What is the default file format for saving in MS Office 2013? Is it still the transitional OOXML or the ISO/IEC 29500 strict format?

As I have read, MS Office 2013 is able to read and write the ISO/IEC 29500 strict (standardized Office Open XML) file format. MS Office 2010 was only able to read the strict format and MS Office 2007 could neither read nor write the strict format. The
.docx, .xlsx and .pptx saved by Office 2007 and 2010 were only complying to the OOXML transitional, as far as I understood.
My main question is what the default file format for saving in MS Office 2013 actually is? Is it still the transitional OOXML or the ISO/IEC 29500 strict format?
In case, MS Office still saves in the transitional standard by default, is it possible to set the strict standard in the settings? Is this recommended (I assume that OOXML strict is the future) to do this?
Thanks.

Yes, the Transitional is still the default file format for saving in Office 2013, and we can modify this setting in Word Options, Save:
As Alex Brown covers in his
blog post, "Transitional is intended to preserve the fidelity of existing binary documents being migrated to ISO/IEC 29500, and includes many legacy features for compatibility with existing documents. Strict is a subset of Transitional that does not
include legacy features – this makes it theoretically easier for a new implementer to support (since it has a smaller technical footprint, so to speak), but also makes it less able to preserve the fidelity of existing documents."
Max Meng
TechNet Community Support

Bank File formats for China

Hello All,
If anyone has worked or come across any of the following for CHINA, then please let us know.
-Bank file formats for incoming/outgoing payments
-Bank statements
-File Formats supported by SAP for China
-Bank Transfers in China
Thanks.
Balla

Wasnt answered.

Can I create File Formats for Bank Transfers?

Hello Everyone!
I need to know if Sap B1 2007A allow me to create File Formats for bank transfers in outgoing payments. I¨m in Colombia, South America, and working as Chile does.
It is available documentation related to this issue?
Thanks very much
Claudia
Edited by: Darius Heydarian on May 6, 2008 5:28 PM
Edited by: Philip Eller on May 20, 2008 9:10 AM

Kerstin,
Thanks for the information.
I might be wrong, but I understood that SAP did allow you to create dll files for this purpose. In the Document Resource Center in the Add-ons section, click on Payment Engine. There is a document called
'Payment Engine Plug-in Development Guide in Release 6.5'.
Claudia - you might want to see if this document is any use for you.
Although the name shows release 6.5, this document still appears for both 2005 and 2007.
I also checked 2007A and it still has the prompt for File Format.
As I said, I might be wrong, because I do not know anyone who has tried to create a plug-in, but if this is not for creating your own dll files, do you know what it is for?
Let me know what you think.
Regards,
Douglas

Fbl file format for Mio Digiwalker C310x?

How can I get NAVTEQ maps converted to fbl file format for Mio Digiwalker C310x? My C310x came with the TeleAtlas maps which are not really great compared to the NAVTEQ maps. I have bought NAVTEQ maps as well, how can I export NAVTEQ maps to the .fbl file format used by iGO/MioMaps?
Post relates to: HP TouchPad (WiFi)

Since this Forum's focus is HP webOS and Palm devices, I'd suggest this question be asked at the Mio Technology forum instead.
WyreNut
I am a Volunteer here, not employed by HP.
You too can become an HP Expert! Details HERE!
If my post has helped you, click the Kudos Thumbs up!
If it solved your issue, Click the "Accept as Solution" button so others can benefit from the question you asked!

Best log file format for multivariable non-continous time series

Similar Messages

Maybe you are looking for