Theory - Best way to store this type of data for retrieval?

Hi everyone,
So I'm pretty new to databases and I'm trying to figure out an efficient way to retrieve this kind of data:
an ID that is 36 digits long
col1 = {option1, option2, option3...}
col2 = {option1, option2, option3...}
coln = {option1, option2, option3...}
I need to be able to perform queries on the data based on some concatenation of 3 digit sequences for the ID
for example, I may want to retrieve all the records where the ID begins with '123' or where the ID begins with '123123', or '123123200', etc,
There are going to be close to a billion if not more of these records
the 3 digit sequences will always be a number between 000-255
It is not necessary for me to have a 36 digit long ID, I simply chose that because it seemed like the simplest way to associate a record with its combination of 3 digit sequences.
Like I said, I'm very new to databases. I've been doing a lot of reading on indexing and clustering and nested tables and partitions, but I'm not quite sure which of these I should pursue with this kind of data. Most of the examples that I've read about don't really deal about querying with variable precision.
Would the best way to get this data simply be to use an 'is like' statement on the ID? If so, are there any kinds of indexing I should take a look at that would benefit from knowing that the is like statements would follow the format of <3 digits,3 digits,3 digits>? Also, at the data level, would it be more efficient to use hexadecimal to store the sequences of 3 digits since they conviently fall in the range of 00-FF? I'm just throwing ideas out there, I haven't found much help browsing the web, but maybe I haven't looked in the right place. If anyone has any ideas or places to redirect me, it would be great!
Also, I know sometimes it helps to know the main kind of queries on the data that someone is primarily concerned about. In this case, most queries will be selective counts, for example finding the number of records whose ID's begin with '123006011' and col1='option1'.
Thanks,
Alex
Edited by: user9022475 on Jun 23, 2010 9:37 AM

So I'm starting to rewrite my code. I was shocked at how easy it was to implement the SAX parser, but it works great. Now I'm on to doing away with my nasty string arrays.
Of course, the basic layout of the XML is like this:
<result1>
<element1>value</element1>
<element2>value</element2>
</result1>
<result2>
I thought about storing each element/value in a HashMap for each result. This works great for a single result. But what if I have 1000 results? Do I store 1000 HashMaps in an ArrayList (if that's even possible)? Is there a way to define an array of HashMaps?

Similar Messages

  • Best way to store, present en input date/time

    Hi!
    I hope I can make myself clear. I'm designing an APEX application that has a couple of different date-type fields. That application is going to be translated in two different languages, that has different date/time representations. For example: dd-mon-yyyy hh12:miAM and dd-mm-yyyy hh24:mi. I'm now finding out what's the best way to store those date/time fields and hou I can present them in a report/form and how the user can enter them in a field.
    This is what I see:
    - I can set an application date format/application timestamp format. This has to be the in the format of the main language, in this case, English.
    - I also can make translated versions of the application. QUESTION: can I enter a different date format for the translated application?
    Do I have to make my fields of the date type or timestamp type?
    I hope someone can answer. If I'm not clear, please tell me!

    You can set the application date format using an application item
    &DATE_FORMAT_MASK.
    and set this item dynamically.
    Home>Application Builder>Application ####>Shared Components>Edit Globalization Attributes
    Denes Kubicek
    http://deneskubicek.blogspot.com/
    http://www.opal-consulting.de/training
    http://apex.oracle.com/pls/otn/f?p=31517:1
    http://www.amazon.de/Oracle-APEX-XE-Praxis/dp/3826655494
    -------------------------------------------------------------------

  • What is the best way to store and search 2D data

    Hi,
    There is a set of data (~10k records ) in 2D dimension.
    like this :
    Col 1, Col 2, Col3....
    What is the best way to store and search those records ?
    Thanks in advance
    Wilson

    Hi,
    Either userObjet[][] if you know how much data you have, and the data size is fixed, or use a list of lists. E.g. A Vector of Vectors (some will probably say that you should use an ArrayList instead, and that could be the case, but it sounds like you would want to display the data later on, and a DefaultTableModel (for JTables) uses a Vector as data holder).
    Kaj

  • Best way to link queue type to data typdef

    I am using "Obtain Queue" to create a multiple queues from multiple TypeDef formated data clusters. I want to then put these queue references in a cluster of their own so I can refer to them where necessary in my code. When I modify the data types by editing the corresponding typdefs I would like for the queue references to change accordingly. In other words I think I need to link the queue references to the data typedefs.  What is the best way to do this?
    For instance, If I create a cluster of queue rerences using bundle by name I would have to create an input cluster with a format that would update as I change my origianl data typedefs. Can this be done, and if so how?
    Possibly the "problem" is due to my dsire  to have a nice "bundle by name" cluster for clarity in my code. I suppose if I used an array this isn't a problem. That is not as elegant a solutions, but I guess elegance is sometimes in the eye of the beholder.
    Any suggestions?
    Solved!
    Go to Solution.

    Did you replace any constants that you had on the block diagram with the typedefed version? All instances that you ar eusing need to use the typedef.
    Mark Yedinak
    "Does anyone know where the love of God goes when the waves turn the minutes to hours?"
    Wreck of the Edmund Fitzgerald - Gordon Lightfoot

  • Best way to store big amount of data

    Hi, i need to store a big amount of data, written in a txt its size is almost 12 mg, anyway it depends on the computer it runs, beacause what i want to store is all the shared files in a computer.
    Which is the best way to store it? Array string? textfile? List? I don�t need the data after the app close.
    Thanks

    Well, then which is the best solution? LinkedList or
    Tree? i only need to store the full path.
    What i didn�t say, my fail, is that i need to search
    for a file name once i have stored them...For searching, LinkedList will be very slow if it's very large. I think the same is true of javax.swing .tree.DefaultTreeModel, which is the JDK's only tree implementation. I don't know what Jakarta-collections has - it's possible they have a tree that offers fast searching. If you want to stick to the standard Java libraries, you'll want a Set for fast searching. TreeSet keeps the entries in sorted order. If you also need to display them as a tree, you can keep them in both a Set and a tree. If you don't have enough memory to do that, then displaying the whole tree isn't going to be useful to the user anyway, so rethink your goal.

  • Best way to store and back up data on Mac (retina)?

    Ok so I bought the 13 inch retina macbook pro (This is my very first mac btw, so I am still trying to get used to basic functions though it's getting there..) and I've researched it a bit but I have some issues on figuring out how I best should store my data.
    1. Is Time Machine not the best way to back up data? As it will delete the oldest data once the hardrive used is full?
    but on the other hand, backing things up manually will take a lot of pain and effort (Choosing the new files that haven't been transferred, accidentally copying a file you actually already put in your back up disk thus taking additional unnecessary space,etc) ...right?
    or is there a better way to do this?
    2. I want to have all media (music, photos, videos, anything except apps) on an external hard drive but I don't want to have to keep transferring it from iphotos or anything else to my external hard drive...
    Is there a way to do this a smarter way? It would be best if I could put everything directly to my harddrive instead of having to import thru iphotos (thus taking up space on my internal SSD first) THEN deleting it from iphotos :/
    As of now I am trying do configure this network storage system (called SeaGate GoFlex Home if anyone is interested) but I am wondering if this is really a reliable way to keep my stuff... I plan on putting music, photos and videos on there..along with word documents
    Is there a way to configure the mac to automatically download stuff to my external network drive instead of saving it on my SSD?
    Thanks in advance for any help /opinions

    kj8P wrote:
    Is Time Machine not the best way to back up data? As it will delete the oldest data once the hardrive used is full?
    Although Time Machine does delete oldest data when full, it doesn't back everything up every time - only the files that have changed since the last backup.  Storage is cheap these days so if you get a good sized drive (seems daft to get less than 1Tb and 2 or 3Tb cost not a great deal more), then unless you have a really large SSD on your MBP and fill it with stuff, it will be a long time before Time Machine starts deleting old stuff. Aim for a minimum of double your SSD capacity.
    This article gives a very detailed explanation of Time Machine and how it works - bear in mind that even when TM has to delete the oldest backups, it will not delete something that does not have a newer version.
    http://icanhelpyourmac.com/tip_archive/understanding_apples_time_machine/index.h tml

  • Best way to store large amounts of data

    Greetings!
    I have some code that will parse through XML data one character at a time, determine if it's an opening or closing tag, what the tag name is, and what the value between the tags is. All of the results are saved in a 2D string array. Each parent result can have a variable number of child results associated with it and it is possible to have over 2,000 parent results.
    Currently, I initialize a new string that I will use to store the values at the beginning of the method.
    String[][] initialXMLValues = new String[2000][45]I have no idea how many results will actually be returned when the method is initially called, so I don't know what to do besides make initialXMLValues around the maximum values I expect to have.
    As I parse through the XML, I look for a predefined tag that signifies the start of a result. Each tag/value that follows is stored in a single element of an arraylist in the form "tagname,value". When I reach the closing parent tag, I convert the arraylist to a String[], store the size of the array if it is bigger than the previous array (to track the maximum size of the child results), store it in initialXMLValues[i.] (<- peroid to avoid post showing up in italics), then increment i
    When I'm all done parsing, I create a new String String[][] XMLValues = new String[i][j], where i is equal to the total number of parent results (from last paragraph) and j is equal to the maximum number of child results. I then use a nested for loop to store all of the values from initialXMLValues into XMLValues. The whole point of this is to minimize the overall size of the returned String Array and minimize the number of null valued fields.
    I know this is terribly inefficient, but I don't know a better way to do it. The problem is having to have the size of the array initilized before I really know how many results I'm going to end up storing. Is there a better way to do this?

    So I'm starting to rewrite my code. I was shocked at how easy it was to implement the SAX parser, but it works great. Now I'm on to doing away with my nasty string arrays.
    Of course, the basic layout of the XML is like this:
    <result1>
    <element1>value</element1>
    <element2>value</element2>
    </result1>
    <result2>
    I thought about storing each element/value in a HashMap for each result. This works great for a single result. But what if I have 1000 results? Do I store 1000 HashMaps in an ArrayList (if that's even possible)? Is there a way to define an array of HashMaps?

  • Best way to store small amount of data to file?

    If I need to store, edit, and retrieve small amounts of data for a desktop app (let's say a small address book with name, address, phone, etc), what are my choices with JavaFX? As far as I can tell, there is no built-in database to handle this...
    What about the same need for a web-based app?
    thanks

    You can use with Netbeans Composer a DataSource and manipulate items with the Query Langage.
    Link

  • Best way to store a LOT of data?

    Hi guys,
    I have a program which contains two arrays of several objects(ie Objects containing other Objects) which I need to store. The arrays can be anywhere from 1000-7000 objects.
    I have used xstream to write this data to disk and I get a file size of about 300mb, which is about 1:1 ratio compared to having the object in memory at runtime.
    Does anyone know of any better algorithms to reduce to file size or is this impossible?
    Thanks!

    Alternatively, you could just use a GZip output stream to compress it on the way out. It would slow you down a bit, though.
    http://java.sun.com/j2se/1.5.0/docs/api/java/util/zip/GZIPOutputStream.html
    Example of XStream w/ GZipOutputStream:
    http://permalink.gmane.org/gmane.comp.java.xstream.user/692
    Message was edited by:
    kevjava

  • Best way to store all this data!!

    Hi i have run into a problem and cant think at the mo. I need to know the best way to store this data:
    Meet(List of events in meet, Runners in meet, Relay team in meet, Which runner/ relay team is registered for which event, results of each event updated once meet is run, event being run(or has finished) or not started yet so runners can still register for it);
    Runner(RunnerID, Name, D.O.B, national team affiliation, Gender, Events in?, Seedtime for each event?, actual time for event?)
    RelayTeam(RelayID, Team name, Runner names, seedtime for events, events in, Acutal time)
    Event(ID, Name, Meet in, Runners in which lane)
    I just want to know how i can store these to make it as easy as possible to update etc.
    Anyhelp apprec.
    Thanks
    J

    Crossposted: http://forums.java.sun.com/thread.jsp?thread=509544&forum=31&message=2419334

  • What is the best way to store a battery?

    Hi,
    I have an extra MBP battery. What is the best way to store this extra battery? Should I store it when it is fully charged? or Should I store it when it is half full? Is it necessary to use that batter once in a while?
    Thank you beforehand.
    Bob

    http://www.apple.com/batteries/notebooks.html
    If you're storing it for a long time, put it away when it's about half full. If you have a couple of batteries for your MacBook, I'd recommend swapping batteries once a month or so to extend the life of both batteries.

  • Best way to store a text document

    I have many (100+) contracts of about 100 pages each. What is the best way to store this data so I can search for text within a contract and search for text across contracts? Filestream? I think I want to be able to do a full text search for strings
    within a documents and across documents..
    Thanks in advance....
    Rich

    Hi,
    If you are using SQL Server 2012, I recommend you to use the FileTable to store the text documents. The FileTable feature of SQL Server 2012 is an enhancement to the FILESTREAM feature which was introduced in SQL Server 2008, FileTable lets
    an application integrate its storage and data management components, and provides integrated SQL Server services - including full-text search and semantic search. Before using
    the FileTable feature, you need to
    enable the FILESTREAM feature on the instance of SQL Server 2012. In addition, you need to create UNIQUE index and FULLTEXT index for the FileTable in order to do full text search.        
    About how to use the FileTable, you can review the following links:
    http://www.mssqltips.com/sqlservertip/2667/filetable-feature-in-sql-server-2012/
    https://www.simple-talk.com/sql/database-administration/full-text-searches-on-documents-in-filetables/
    Thanks                 
    Lydia Zhang

  • Best way to store an AES Key

    Hello,
    I'm currently developing a little "file vault" , with graphical interface, to enable end users to encrypt/decrypt some of their personal files on teh filesystem
    I'm currently using AES as Cipher :
    the first time the user launches the UI, i generate an key for AES then store it (encoded as Base64) in Registry (if Windows) using java.util.Preferences
    Doing this, later, the application automatically retrieves this key for decrypt/encrypt tasks.
    Is this the best way to store this kind of key.
    I currently ask myself if i couldn't (as second option) leave the user choose a key first : the user should then retype this key at every launch of the application to be able to decrypt/encrypt his files.
    What is the best way?
    Laurent

    As far as i understand in the document PKCS#5 , to do my application, i should use the PBES2 (password based encryption scheme 2), respecting the following steps :
    Encryption:
    1) Select a Password (P) : this password is not stored on the system, the user will provide it to the application for every operation (dec / enc)
    2) Select a Salt (S) : i wonder if first time i should use a pseudo random generator, then store the Salt on the System (where on Windows or Linux?...)
    3) Select an iteration Count (c) : should be > 1000
    4) select a derived key Length (dkLen)
    5) Create a derived key (DK) using the formula : DK = DKF(P,S,c,dkLen)
    6) Encrypt clear Message M using the Derived Key (DK) an the algorithm AES, to obtain the Cipher Text (C)
    Decryption:
    Repeat steps 1 to 5
    6) Decrypt Cipher Text (C) using DK and AES algorithm to obtain the clear message (M)
    Does it sound good for you.
    My question is always the same :
    Should i hard code in my application the Salt ,the Iteration Count, and the derived key length or , should i first generate a pseudo random Salt, then store it on the file system (Windows Registry, File)
    Any idea?
    Laurent
    Message was edited by:
    Laurent_Bois

  • What is the best way to store data for this project?

    hey everyone,
    I have been subscribed to this for a while, not sure if I have ever actually asked anything though.
    I have a project on the go for myself/portfolio
    It is a booking sheet, where by I have a GUI that has a diary system of a day followed by time slots. This also has a date picker that can change the date of the booking sheet
    I want to be able to store mainly strings and ints,
    I need to be able to store, retrieve and on occasion change some data.
    I was looking at using something called JExcelAPI but I cant get that to work at all, I asked for assistance but was refered to here.
    what would be the best way to implement this data storage?
    davyk

    Hey everyone,
    Back again,
    I got this this little snippet of code working but want to ask you guys for a little bit of help on it. if thats ok?
    try
         Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
         String dataSourceName = "mdbTEST";
         String dbURL = "jdbc:odbc:" + dataSourceName;
         Connection con = DriverManager.getConnection(dbURL, "","");
         // try and create a java.sql.Statement so we can run queries
         Statement s = con.createStatement();
         s.execute("create table TEST1234567 ( column_1 char(27), column_2 char(150), column_3 char(150), column_4 char(150), column_5 char(150), column_6 char(150))"); // create a table
         s.execute("insert into TEST1234567 values('"+date+"','"+a+"','"+b+"','"+c+"','"+d+"','"+e+"',)"); // insert some data into the table
         s.execute("select column_7 from TEST1234567"); // select the data from the table
         ResultSet rs = s.getResultSet(); // get any ResultSet that came from our query
         if (rs != null) // if rs == null, then there is no ResultSet to view
                    while ( rs.next() ) // this will step through our data row-by-row
              /* the next line will get the first column in our current row's ResultSet
              as a String ( getString( columnNumber) ) and output it to the screen */
                   System.out.println("Data from column_2: " + rs.getString(1) );
         s.execute("drop table TEST1234567");
         s.close(); // close the Statement to let the database know we're done with it
         con.close(); // close the Connection to let the database know we're done with it
    catch (Exception err)
         System.out.println("ERROR: " + err);
         err.printStackTrace();
    }there are more columns, but i cut this code down.
    my question is:
    I think I want a method with an if statement to see whether the table is created or not, if not create it, but how do I go about this? I have searched the API and google, but my brain is fried.
    Also do I always have to do the try/catch and have the code of Class.forname to Statement s in all methods that want to deal with the table?
    davy

  • HT1451 I need to move my library from my old computer to my new laptop. What's the best way to do this?

    I need to move my library from my old computer to my new laptop (Windows 8). What's the best way to do this?

    These are two possible approaches that will normally work to move an existing library to a new computer.
    Method 1
    Backup the library with this User Tip.
    Deauthorize the old computer if you no longer want to access protected content on it.
    Restore the backup to your new computer using the same tool used to back it up.
    Keep your backup up-to-date in future.
    Method 2
    Connect the two computers to the same network. Share your <User's Music> folder from the old computer and copy the entire iTunes library folder into the <User's Music> folder on the new one. Again, deauthorize the old computer if no longer required.
    Both methods should give the new computer a working clone of the library that was on the old one. As far as iTunes is concerned this is still the "home" library for your devices so you shouldn't have any issues with iTunes wanting to erase and reload.
    I'd recommend method 1 since it establishes an ongoing backup for your library.
    If you have an iOS device that syncs with contact & calendar data on your computer you should migrate this information too. If that isn't possible create a dummy entry of each type in your new profile and iTunes should offer to merge the existing data from the device into the computer, otherwise the danger is that it will wipe the information from the device.
    If your media folder has been split out from the main iTunes folder you may need to do some preparatory work to make it easier to move. See make a split library portable.
    Should you be in the unfortunate position where you are no longer able to access your original library, or a backup of it, then see Recover your iTunes library from your iPod or iOS device for advice on how to set up your devices with a new library with the maximum preservation of data. If you don't have any Apple devices then see HT2519 - Downloading past purchases from the App Store, iBookstore, and iTunes Store.
    tt2

Maybe you are looking for