Using a textfile to store ~300,000 - 1million records of data

Hi all,
I'm making a simple java program that will retrieve information, store it. Then the application would use the stored information.
My thought was -- in order to make the java program work on all platforms, rather than use a database, i would just save the records of information to a txt file.
Imagine a file holding the following information for about 300,000 - 1 million rows.
prop1=value1, value2, value3
prop2=value1a, value2a, value3a
prop3=value1b, value2b, value3b
prop4=value1c, value2c, value3c
prop5=value1d, value2d, value3d
later on the application would load these retrieved records as a property object.
Do you guys think it would work or do you think I'm better off using a database? on the windows platform I would be using access 2000
Thanks for the thoughts in advance.
stev

I would have to go with the database on this.
For the given data, each record is probably 10 to 1000 bytes of data.
Unless you build an index, any search is going to take a bit of time with 10 bytes and significantly more with 1000.
If you try to keep it in memory even for the smallest size it would take (4+4+10)*records plus some overhead, so 5.4 meg (300,000 records). And that is using a packed format and is probably optmistic on the memory usage.
At the upper range it simply won't fit (4+4*1000)*1,000,000 is 1.8 gig. The Sun 1.3.1 JVMs for solaris and windows do not support that much memory.
That isn't to say that you couldn't keep it as a file. But to do so would require a non-trivial amount of code to efficiently access it. And without more information about why the database is not a good solution (or how the app would be used) it is hard to say what the best solution is.

Similar Messages

Exporting and Importing Large data set of approx 300,000 rows

Hi,
I have a table in db 1(approx 10 columns) and want to copy all the rows(approx. 300,000) from this table with a simple where clause to the same db table in db 2. Both databases are on unix.
I am executing this from a laptop with windows XP remotely connected to my office network.
Could someone let me know what is the best way to do this.
Thanks

331991 wrote:
Thanks for the detailed instructions. I however have some limitations:
1. Both schemas(from and to) are on the same database server. I am the schema owner for both the schemas A logical impossibility in Oracle. A schema is defined by its owner, therefor you cannot have two schemas owned by the same owner. SCHEMA1 is, by definition, owned by user SCHEMA1. And SCHEMA2 is, by definition, owned by user SCHEMA2.
but don't have rights to create dir's on the server.Then it is of no value to create a directory object in the database. The database's directory object is nothing but an alias to refer to an actual, existing directory on the host server.
2. I don't want to copy all data from db1 to db2. I want to copy data where id > 5000 in the id column. Also in table in db1 has 10 columns and table in db2 has 15 columns with 10 from db1 with same data types and 5 more .
How can I export the data to my laptop from db1 and then import into db2 in this scenario?
You don't. Create a dblink and copy the data directly from the source to the target:
INSERT INTO TARGET_TABLE VALUES
(SELECT COLA, COLB, COLC, NULL, NULL, NULL, SYSDATE
FROM SOURCE_TABLE@SOURCE_LINK
WHERE ID > 5000);
Also I am using Oracle 10G.
Thanks

1. I have an iPod which has purchases synced with I tunes with an apple ID. I now have an iPad with a new ID for cloud. I have cloud storage capacity of 5gb which came with the iPad. How can I use one ID and store my existing music in cloud?

1. I have an iPod which has purchases synced with I tunes with an apple ID. I now have an iPad with a new ID for cloud. I have cloud storage capacity of 5gb which came with the iPad. How can I use one ID and store my existing music in cloud?

HI Frostyfrog
CHeck out iTunes Match which has an annual fee of £21 or thereabouts. You can store a maximum of 25,000 songs there and they all become available on your other kit, iPhone, iPad and iPod. I know it works as I have over 23000 songs uploaded of which only a few we're bought through iTunes.
it works by comparing your music to the whole iTunes music dadata base so you access the same tunes that you could get from iTunes. If you have obscure stuff, it uploads your own music to the cloud as a copy.
I Find it incredible that on my iPhone with its 16gb memory I can view almost 200 gb of music (ie my 23000 songs) and play any of them. Anything I add to iTunes becomes available via the cloud fairly quickly from a few minutes or a little longer if adding a lot.
The first time you use it it will take quite a while to match your music if you have a lot, but after that it is all automatic. Read the stuff on the apple site. Your PC needs to be at least running Vista. I recommend it and at less than 50p a week it's good value.
Good luck

Notes Multiply alarmingly - from 10 Notes to 300,000

Mail has been really problematic foe me recently (refuses to Quit unless ForceQuit, hogs up all CPU, is sluggish in general and clicking on Notes and ToDos crashes Mail.)
I have tried various remedies (rebuilt Mailboxes, trashed Preferences, stopped syncing Notes) but nothing has helped.
I looked in the ~/Library/Mail/Mailboxes/Notes.mbox/Messages directory and found over 300,000 items when I have only about 10 notes. There are multiple copies of these 10 Notes in that folder. And the number of items in that folder increases as long as Mail is running. The size of this folder is now 1.35 GB when each Note is 8KB or less. As soon as I start Mail these begin to multiply.
I am using MobileMe but have turned off Notes Syncing. Even then the Notes seem to multiply on their own.
I have 2 IMAP and one POP folder.

VK,
Thanks for your help. I tried the steps you mentioned (deleted the Notes.mbox and plist files, recreated the accounts) but the Notes kept multiplying. Finally, in exaspration I just deleted all the Notes and the multiplying stopped. HOWEVER, despite removing all the Notes, recreating the Mailboxes and unchecking Notes sync, Mail still crashes when I click on Notes.
In addition, even after all those changes (and many many more prior to that) Mail is horribly slow, it periodically consumes all the CPU making my (otherwise very well-behaved) laptop extremely hot with the fans blowing REALLY hard and refuses to Quit unless Force Quit. It may be time for me to go back to Eudora. All of this started happening since 10.5.4 or the MobileMe update.

Script logic record more than 300,000 record

Hi Expert,
When I run my logic I have error in my formula log:
(More than 300,000 records. Details are not being logged)
Ignoring Status
Posting ok
I check my script it pull out total 422076 records.
Is it meaning I cannot More than 300,000 records??
Is there any where I can set MAX records I can generate for my single script to run??
Thanks..

You should use
*XDIM_MAXMEMBERS dimension = numberOfMembers to be processed at a time
For example
*XDIM_MAXMEMBERS Entity = 50
Figure out wich dimension has the most members, and use it, this sections you script logic.
I hope that helps
Leandro Brasil

BKPF 6000 records and bsid 300,000 records

Which one is better ?
1.
LOOP bkpf (6000 records )
         READ bsid (300,000 records )
         READ .....
         READ .....
         READ .....
         READ .....
   ENDLOOP.
and the other thing that i concern in (1.) is perhap i have to use
LOOP bkpf (6000 records )
         loop bsid (300,000 records ) where .....
   ENDLOOP.
2.
LOOP bsid(300,000 records)
       READ bkpf(300,000 records )
   ENDLOOP.
Now in my program i use (2.) but performance is quite bad. it's time out (1 hour) on PRD
actually i have many internal tables to read not only bkpf but it's not my concern and all of READ i SORT and use BINARY SEARCH
Thank you in advance

Try the below code.
sort it_bsid by bukrs belnr gjahr.
loop at it_bkpf.
clear: lv_indx.
read table it_bsid with key bukrs = it_bkpf-rbukrs
                                         belnr = it_bkpf-belnr
                                         gjahr = it_bkpf-gjhar
                                         binary search.
if sy-subrc = 0.
    lv_indx = sy-tabix.
    loop at it_bsid from lv_indx.
      if it_bsid-bukrs = it_bkpf-bukrs and
         it_bsid-belnr = it_bkpf-belnr and
         it_bsid-gjahr = it_bkpf-gjahr.
        << read other internal tables and do the necessary processing>>
      else.
         clear: lv_indx.
         exit.
      endif.
    endloop.
endif.
endloop.
Hope this helps your time out issue.
Thanks,
Balaji

Ekpo-netpr is 83,000,000.00 but the screen value is 8,300,000.000

the po price display 8,300,000.000 via me23n, but in table, the value of ekpo-netpr is 83,000,000.00.
the two value is not same, how can i get the correct po price.

unfortunatly you didnot tell the currency.
If it is a currency with 0 decimals like Yen then the value in table is always stored with 2 decimals according to the domain of the currency field but displayed using the decimlas from the currency table.
from OSS note 1176399:
SAP format for amounts
In an SAP system, amounts (data type CURR) are always stored in the database using exactly two decimal places. It does not matter how many decimal places are actually allowed for the relevant currency.
However, amounts are displayed on screens as converted in accordance with their number of decimal places. The number of decimal places, and therefore the shift of the decimal point, is stored in the table TCURX.
Example
The currencies JPY (Japanese Yen) and TWD (Taiwanese Dollar) do not have any decimal places. An amount of 123456 JPY is stored on the database as 1234.56 JPY; however, on the screen (for example, in the PSA maintenance, in the query or in the LISTCUBE) it is displayed as 123,456 JPY (when using the comma as the thousand separator).

How do I switch my apple Id I use in the iTunes store to the one I have on iCloud?

how do I switch my apple Id I use in the iTunes store to the one I have on iCloud?

You can simply sign out and sign in with the different ID. However, if you have any protected media - movies, or songs purchased before they stopped protecting them - you will not be able to play them, so in that case you are rather stuck.
You can't merge accounts or transfer iTunes purchases from one Apple ID to another: but there is no problem about using two IDs. Use your iCloud ID for iCloud for syncing email, contacts, calendars, bookmarks, and PhotoStream; use your other Apple ID as before for iTunes, iTunes in the Cloud and iTunes Match.
There's no conflict about doing this, and you won't even notice as Keychain will log you in with the correct ID in each case. It's also better security not to have your iTunes login also your email address, given the number of complaints about hacked iTunes accounts.

How do I use 2 different iTunes Stores?

How do I use 2 different iTunes Stores simultateously? Active account in South Africa, but need to download apps from the German iTunes Store - which iTunes does not allow, as these apps do not appear on the South African iTunes Store.

You cannot do this.

Help! My iPod touch won't let me download any apps. A message keeps coming up saying "This Apple ID cannot be used for the iTunes store. Please enter another Apple ID" I changed my ID to see if that would work but the same message still comes up.

Help! My iPod touch won't let me download any apps. A message keeps coming up saying "This Apple ID cannot be used for the iTunes store. Please enter another Apple ID" I changed my ID to see if that would work but the same message still comes up.

Where are you physically located?
What country's Store are you using?
The two have to be the same and the payment method for your account also has to be associated (billing address, where iTunes gift card was purchased) with that country.
Has the ID(s) worked before for purchases for this country's store?

Why when i want to download a free app from app store it asks for my apple id and when i give my apple id it then says "this apple id has not yet been used in the itunes store". i dont even know what that means and it will this happen everytime?

why when i want to download a free app from app store it asks for my apple id and when i give my apple id it then says "this apple id has not yet been used in the itunes store". i dont even know what that means and it will this happen everytime?

If you are within 14 days of purchase you can still return the iPhone and get the phone that suits your needs, however, I think you will find that no matter which phone you get the content provider will want to know how you are going to pay for future purchases. For Android phones it is Google, and they will certainly want to know.
To create an Apple account without a credit card see: http://support.apple.com/kb/HT2534

I have a iphone 5 and I can login with my apple id to purchase music. However, when I try to login into icloud using the very same username and password that I use in the apple store it does not work to enter icloud, so what what gives???

I have a iphone 5 and I can login with my apple id to purchase music. However, when I try to login into icloud using the very same username and password that I use in the apple store it does not work to enter icloud, so what what gives???

I could do that, however when I select the icloud button (or whatever the heck it is) I am asked to enter the apple id and password. So if you are suppose to create another one for icloud you'd think it would give you the option at this point which would be logical.

When i log in my apple id in itunes a message prompt me "This Apple ID has not yet been used with the itunes store" what do i need to do?

when i log in my apple id in itunes a message prompt me "This Apple ID has not yet been used with the itunes store" what do i need to do?

If you can log into the computer, go to the Mac App Store and purchase Mavericks under your AppleID. Even though its free, they still consider it a purchase.
If you are not able to log into your Mac you can try on a different Mac. Once you purchase it, you can stop the download if you want.
Once you do that then you should be able to reinstall via the Recovery HD

I cannot sign in. "This Apple ID has not yet been used with the iTunes Store. Please review your account information." I click "Review", next message "You must turn on cookies to continue.   Cookies must be enabled if you want to create or edit.. etc."

I applied and received an upgrade code to Redeem Mountain Lion for my new MBP.
I get this message when I try to Sign in to the App Store after entering the redemption content code, and also if I just try to "Sign In".   The first part of the error is true - I haven't used the AppleID with the iTunes store.
I received the upgrade code using this same AppleID, so I don't think I can use a different AppleID.
I also tried to re-create this appleID...(it was one suggestion I saw for this issue) It doesn't let me since the ID already exists...
The error is:
"This Apple ID has not yet been used with the iTunes Store. Please review your account information." I click "Review", next message "You must turn on cookies to continue.   Cookies must be enabled if you want to create or edit.. etc."
It's clearly not a cookie issue..   Stuck...

The first part of the error is true - I haven't used the AppleID with the iTunes store.
Try here > Using an existing Apple ID with the iTunes Store and Mac App Store
received the upgrade code using this same AppleID, so I don't think I can use a different AppleID.
Correct.
It's clearly not a cookie issue..
As for cookies. Quit the App Store. Now open the Finder. From the Finder menu bar click Go > Go to Folder.
Type this exactly as you see it here:   ~/Library/Cookies
Click Go the move the   com.apple.appstore.plist   file from the Cookies folder to the Trash.
Relaunch the App Store.

I have a Apple ID, however when i am trying to login to App Store, i am getting a message "This Apple ID has not yet been used with the App Store"?

Hello,
I have a Apple ID, however when i am trying to login to App Store or iTunes, i am getting a message "This Apple ID has not yet been used with the App Store"?
I am able to login to iCloud with my Apple ID
Kindly Assist
Regards
Shuja

If you have never used the Apple ID to make purchases from iTunes or the Mac App Store you need to follow the steps to create an account with that Apple ID.

Using a textfile to store ~300,000 - 1million records of data

Similar Messages

Maybe you are looking for