Best way to store a LOT of data?

Hi guys,
I have a program which contains two arrays of several objects(ie Objects containing other Objects) which I need to store. The arrays can be anywhere from 1000-7000 objects.
I have used xstream to write this data to disk and I get a file size of about 300mb, which is about 1:1 ratio compared to having the object in memory at runtime.
Does anyone know of any better algorithms to reduce to file size or is this impossible?
Thanks!

Alternatively, you could just use a GZip output stream to compress it on the way out. It would slow you down a bit, though.
http://java.sun.com/j2se/1.5.0/docs/api/java/util/zip/GZIPOutputStream.html
Example of XStream w/ GZipOutputStream:
http://permalink.gmane.org/gmane.comp.java.xstream.user/692
Message was edited by:
kevjava

Similar Messages

Best way to store a lot of image files??

I am working on a new project which involve user upload image to server and then do searching and view image. there will be a lot of image files. image will be displayed in web page as thumbnails, so there may be resized pic stored in server or resize on the fly.
What is the best way to store image like this? as files in file system? database? etc...?
Thank you very much

Since seraching is invloved use a database. A file system is not good place to search in.
Using a database will allow you to store additional data as well such as comments about the images or the ratings given by the users.
Of course you will have to develop a propper table structure for this.
Also do not generate thumbnails all the time instead cache them once they are created. Atleast the thumbnails of the images that are displayed often.
Also store the last updted dates with the images so you can loow the images to be cached by the clients or the proxy server. That will reduce repatitive requests for the same image from same client.

What is the best way to store and search 2D data

Hi,
There is a set of data (~10k records ) in 2D dimension.
like this :
Col 1, Col 2, Col3....
What is the best way to store and search those records ?
Thanks in advance
Wilson

Hi,
Either userObjet[][] if you know how much data you have, and the data size is fixed, or use a list of lists. E.g. A Vector of Vectors (some will probably say that you should use an ArrayList instead, and that could be the case, but it sounds like you would want to display the data later on, and a DefaultTableModel (for JTables) uses a Vector as data holder).
Kaj

Best way to store big amount of data

Hi, i need to store a big amount of data, written in a txt its size is almost 12 mg, anyway it depends on the computer it runs, beacause what i want to store is all the shared files in a computer.
Which is the best way to store it? Array string? textfile? List? I don�t need the data after the app close.
Thanks

Well, then which is the best solution? LinkedList or
Tree? i only need to store the full path.
What i didn�t say, my fail, is that i need to search
for a file name once i have stored them...For searching, LinkedList will be very slow if it's very large. I think the same is true of javax.swing .tree.DefaultTreeModel, which is the JDK's only tree implementation. I don't know what Jakarta-collections has - it's possible they have a tree that offers fast searching. If you want to stick to the standard Java libraries, you'll want a Set for fast searching. TreeSet keeps the entries in sorted order. If you also need to display them as a tree, you can keep them in both a Set and a tree. If you don't have enough memory to do that, then displaying the whole tree isn't going to be useful to the user anyway, so rethink your goal.

Best way to store, present en input date/time

Hi!
I hope I can make myself clear. I'm designing an APEX application that has a couple of different date-type fields. That application is going to be translated in two different languages, that has different date/time representations. For example: dd-mon-yyyy hh12:miAM and dd-mm-yyyy hh24:mi. I'm now finding out what's the best way to store those date/time fields and hou I can present them in a report/form and how the user can enter them in a field.
This is what I see:
- I can set an application date format/application timestamp format. This has to be the in the format of the main language, in this case, English.
- I also can make translated versions of the application. QUESTION: can I enter a different date format for the translated application?
Do I have to make my fields of the date type or timestamp type?
I hope someone can answer. If I'm not clear, please tell me!

You can set the application date format using an application item
&DATE_FORMAT_MASK.
and set this item dynamically.
Home>Application Builder>Application ####>Shared Components>Edit Globalization Attributes
Denes Kubicek
http://deneskubicek.blogspot.com/
http://www.opal-consulting.de/training
http://apex.oracle.com/pls/otn/f?p=31517:1
http://www.amazon.de/Oracle-APEX-XE-Praxis/dp/3826655494
-------------------------------------------------------------------

Best way to store and back up data on Mac (retina)?

Ok so I bought the 13 inch retina macbook pro (This is my very first mac btw, so I am still trying to get used to basic functions though it's getting there..) and I've researched it a bit but I have some issues on figuring out how I best should store my data.
1. Is Time Machine not the best way to back up data? As it will delete the oldest data once the hardrive used is full?
but on the other hand, backing things up manually will take a lot of pain and effort (Choosing the new files that haven't been transferred, accidentally copying a file you actually already put in your back up disk thus taking additional unnecessary space,etc) ...right?
or is there a better way to do this?
2. I want to have all media (music, photos, videos, anything except apps) on an external hard drive but I don't want to have to keep transferring it from iphotos or anything else to my external hard drive...
Is there a way to do this a smarter way? It would be best if I could put everything directly to my harddrive instead of having to import thru iphotos (thus taking up space on my internal SSD first) THEN deleting it from iphotos :/
As of now I am trying do configure this network storage system (called SeaGate GoFlex Home if anyone is interested) but I am wondering if this is really a reliable way to keep my stuff... I plan on putting music, photos and videos on there..along with word documents
Is there a way to configure the mac to automatically download stuff to my external network drive instead of saving it on my SSD?
Thanks in advance for any help /opinions

kj8P wrote:
Is Time Machine not the best way to back up data? As it will delete the oldest data once the hardrive used is full?
Although Time Machine does delete oldest data when full, it doesn't back everything up every time - only the files that have changed since the last backup. Storage is cheap these days so if you get a good sized drive (seems daft to get less than 1Tb and 2 or 3Tb cost not a great deal more), then unless you have a really large SSD on your MBP and fill it with stuff, it will be a long time before Time Machine starts deleting old stuff. Aim for a minimum of double your SSD capacity.
This article gives a very detailed explanation of Time Machine and how it works - bear in mind that even when TM has to delete the oldest backups, it will not delete something that does not have a newer version.
http://icanhelpyourmac.com/tip_archive/understanding_apples_time_machine/index.h tml

Theory - Best way to store this type of data for retrieval?

Hi everyone,
So I'm pretty new to databases and I'm trying to figure out an efficient way to retrieve this kind of data:
an ID that is 36 digits long
col1 = {option1, option2, option3...}
col2 = {option1, option2, option3...}
coln = {option1, option2, option3...}
I need to be able to perform queries on the data based on some concatenation of 3 digit sequences for the ID
for example, I may want to retrieve all the records where the ID begins with '123' or where the ID begins with '123123', or '123123200', etc,
There are going to be close to a billion if not more of these records
the 3 digit sequences will always be a number between 000-255
It is not necessary for me to have a 36 digit long ID, I simply chose that because it seemed like the simplest way to associate a record with its combination of 3 digit sequences.
Like I said, I'm very new to databases. I've been doing a lot of reading on indexing and clustering and nested tables and partitions, but I'm not quite sure which of these I should pursue with this kind of data. Most of the examples that I've read about don't really deal about querying with variable precision.
Would the best way to get this data simply be to use an 'is like' statement on the ID? If so, are there any kinds of indexing I should take a look at that would benefit from knowing that the is like statements would follow the format of <3 digits,3 digits,3 digits>? Also, at the data level, would it be more efficient to use hexadecimal to store the sequences of 3 digits since they conviently fall in the range of 00-FF? I'm just throwing ideas out there, I haven't found much help browsing the web, but maybe I haven't looked in the right place. If anyone has any ideas or places to redirect me, it would be great!
Also, I know sometimes it helps to know the main kind of queries on the data that someone is primarily concerned about. In this case, most queries will be selective counts, for example finding the number of records whose ID's begin with '123006011' and col1='option1'.
Thanks,
Alex
Edited by: user9022475 on Jun 23, 2010 9:37 AM

So I'm starting to rewrite my code. I was shocked at how easy it was to implement the SAX parser, but it works great. Now I'm on to doing away with my nasty string arrays.
Of course, the basic layout of the XML is like this:
<result1>
<element1>value</element1>
<element2>value</element2>
</result1>
<result2>
I thought about storing each element/value in a HashMap for each result. This works great for a single result. But what if I have 1000 results? Do I store 1000 HashMaps in an ArrayList (if that's even possible)? Is there a way to define an array of HashMaps?

Best way to store large amounts of data

Greetings!
I have some code that will parse through XML data one character at a time, determine if it's an opening or closing tag, what the tag name is, and what the value between the tags is. All of the results are saved in a 2D string array. Each parent result can have a variable number of child results associated with it and it is possible to have over 2,000 parent results.
Currently, I initialize a new string that I will use to store the values at the beginning of the method.
String[][] initialXMLValues = new String[2000][45]I have no idea how many results will actually be returned when the method is initially called, so I don't know what to do besides make initialXMLValues around the maximum values I expect to have.
As I parse through the XML, I look for a predefined tag that signifies the start of a result. Each tag/value that follows is stored in a single element of an arraylist in the form "tagname,value". When I reach the closing parent tag, I convert the arraylist to a String[], store the size of the array if it is bigger than the previous array (to track the maximum size of the child results), store it in initialXMLValues[i.] (<- peroid to avoid post showing up in italics), then increment i
When I'm all done parsing, I create a new String String[][] XMLValues = new String[i][j], where i is equal to the total number of parent results (from last paragraph) and j is equal to the maximum number of child results. I then use a nested for loop to store all of the values from initialXMLValues into XMLValues. The whole point of this is to minimize the overall size of the returned String Array and minimize the number of null valued fields.
I know this is terribly inefficient, but I don't know a better way to do it. The problem is having to have the size of the array initilized before I really know how many results I'm going to end up storing. Is there a better way to do this?

So I'm starting to rewrite my code. I was shocked at how easy it was to implement the SAX parser, but it works great. Now I'm on to doing away with my nasty string arrays.
Of course, the basic layout of the XML is like this:
<result1>
<element1>value</element1>
<element2>value</element2>
</result1>
<result2>
I thought about storing each element/value in a HashMap for each result. This works great for a single result. But what if I have 1000 results? Do I store 1000 HashMaps in an ArrayList (if that's even possible)? Is there a way to define an array of HashMaps?

Best way to store small amount of data to file?

If I need to store, edit, and retrieve small amounts of data for a desktop app (let's say a small address book with name, address, phone, etc), what are my choices with JavaFX? As far as I can tell, there is no built-in database to handle this...
What about the same need for a web-based app?
thanks

You can use with Netbeans Composer a DataSource and manipulate items with the Query Langage.
Link

Best way to store data from a plot on hard drive

I have a DAQ set-up that will be on 24/7 for several days, and the whole time it's streaming data to an XY-Graph with a limited buffer. My question is, what's the best way to store my plot data in such a way that I can empty out my plot every few hours but if I want to pull up data from a few hours ago I would just have an option on my initial VI to view any old saved plot data.
Basically, I'm trying to not overload my RAM with all my plot data so I'd like to save it to the hard drive while my VI is running, and only display, for example the last 8 hours worth of data... I've been looking at TDMS, but I'm not sure if that's the right solution.
much thanks!

Hi,
You could save your data using TDMS as well as ASCII to simply load it in another application later.
Attached VI shows you how to save your data to a File after a certain amount of iterations, you can combine this then with your code and a timestamp.
Christian
Attachments:
Save to new file every N samples.vi ‏20 KB

Best way to store lots of tile images

Hey, In my game I'm gonna need to store probably thousands of tile images. They are all in .png format, and I am wondering the best way to store the image data. I could of course keep them in some folder called images, and then have thousands of files...but then people could jack them, and plus that would be inneficient use of space??? right?
What is the best way to store them??
thanks

Anything can be "jacked". The other possible ways are storing them in some sort of archived file like a JAR or ZIP or GZIP. The only way to even remotely ensure no one will "jack" your images would be to store them in a compressed format which you created. Which would entail creating your own compression algorithms. Have Fun! :)
Seriously though, I wouldn't worry too much about the images and people "jacking" them. As far as inneficient use of space, I personally wouldn't worry about this either unless your app will be delivered via WebStart or Applet. Then load time could be an issue. You could use lazy downloading though and download the images when they are needed.
HTH
Gregg

Best way to store image? will be a lot of image files

I am working on a new project which involve user upload image to server and then do searching and view image. there will be a lot of image files. image will be displayed in web page as thumbnails, so there may be resized pic stored in server or resize on the fly.
What is the best way to store image like this? as files in file system? database? etc...?
Thank you very much

Hi,
When you are asking about the "best method for BDB", you should first read this: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/select.html
When you know exactly what are the differences between the BDB access methods, then you will be able to figure out which is the best method for your requirements or to ask a more specific question.
Regarding the second problem, you should read about isolation and degrees of isolation: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/inc.html
Regards,
Bogdan Coman

Best way to store all this data!!

Hi i have run into a problem and cant think at the mo. I need to know the best way to store this data:
Meet(List of events in meet, Runners in meet, Relay team in meet, Which runner/ relay team is registered for which event, results of each event updated once meet is run, event being run(or has finished) or not started yet so runners can still register for it);
Runner(RunnerID, Name, D.O.B, national team affiliation, Gender, Events in?, Seedtime for each event?, actual time for event?)
RelayTeam(RelayID, Team name, Runner names, seedtime for events, events in, Acutal time)
Event(ID, Name, Meet in, Runners in which lane)
I just want to know how i can store these to make it as easy as possible to update etc.
Anyhelp apprec.
Thanks
J

Crossposted: http://forums.java.sun.com/thread.jsp?thread=509544&forum=31&message=2419334

What's the best way to store my info externally

I have recently bought my first Macbook Pro 13" with 2.4 Ghz. I am currently using it for my own business and do tend to get a lot of files together. I went to look at "About this Mac" and have already got 369 Gb used up in storage... I have no idea how I have done that as I have hardly downloaded any apps on this computer besides the ones that were on here. I did download Vware Fusion 4 as I have a reservation system which I used to run on my pc and need to have this on here too so perhaps that software takes a lot of space.
Could anyone advise on which is the best way to store my info externally, will any external harddrive do, or do you need a special one for Apple or Macbook pro. I am just asking as I have no idea and prefer to buy the right one straight away.
Any help would be much appreciated !
Thanks, Sara***

Sara,
The most reliable method is to use an external hard drive. Some are better than others, but most will work.
For the least amount of setup pain, I prefer to deal with the Mac specialists at OWC, http://www.macsales.com as they stand behind their products with great warranty and support. I have one of their Mercury Elite Pro 2 TB drives that has multiple interface capability so you can get a fast interface for data transfer.
Take a look at what they offer and see what fits your needs. If you have more questions about doing thin, just ask on here.
Ralph

Best way to store documents and images in a web application

All,
Which is the best way to store the documents, is it Oracle DB or file system or Virtual Directory Mapping in weblogic.
We are working on a Internet based application into CRM domain, there is a requirement where user can upload and download the documents (.pdf, .xls, .doc etc). Right now we are storing images in the DB. Is it advisable to store all the docs in database and will it scale up in future.
PS: There is no scope for using content server because of business constraints.
Please let me know, thanks in advance.
Regards
Girish

Best way to store depends entirely on requirements.
Why store it in the database when it for example is purely static images (e.g. used as part of the web GUI)? Instead that can be served up directly by a web server - which is specifically designed for this.
If it is more than just an image, but include tags (non spatial data attributes), requires transaction control (e.g. updating an old passport/id photo with a newer one), auditing, filtering/searching and ordering, requires backup, needs security and access controls, and so on - than that is most definitely a database function.
So the issue is not what the binary file is or contains - an image or pdf or something else. It depends entirely on how it is to be used and what the business requirements are in this respect.

Best way to store a LOT of data?

Similar Messages

Maybe you are looking for