Exception while Read very large file 300 MB

Hi
I want to read a file which has size of more than 300 MB.
While i m executing my code, i m getting
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Below is my code
1. FileChannel fc = new FileInputStream(fFile).getChannel();
2.CharBuffer chrBuff = Charset.forName("8859_1").newDecoder().decode(fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size()));
3.fc.close();Im getting the exception in the line 2.
Even though i increased the Heap space up to 900 Mb, i m getting this error.
(FYI)
I executed in command prompt like below,
java -Xms128m -Xmx1024m readLargeFile
Kindly give a solution for this. is there any other best way to read large text file ??
I m waiting for your reply..
Thanks in advance

Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.
The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.
Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.
Thanks in Advance...

Similar Messages

  • How do I control read start position in a very large file where start byte position may be larger than I32 (+/- 2^31)?

    Using LabView, I am trying to read a very large file which may be on the order of 2^32 bytes. I need to be able to step into the file at a byte position which may be greater than the I32 limit set by the read file.vi. Are there any options to the read file.vi or a method of circumventing this limitation?

    I'm not sure but i think that you can manage the "pos mode" in the "seek" sub-vi.
    The "pos mode" let you choose the initial position to add the numbers of bytes you want to move.
    I think that you can add a i32 number from the "initial" in the "pos mode" an latter use the "pos mode" in "current" to add another value. Then the next time you can move more than 2^31 bytes to the initial position.
    I hope you understand my idea, i wasn't try it before, but i think that would work.

  • Exception thrown while reading an XML file using Properties

    Exception thrown while reading an XML file using Properties.
    Exception in thread "main" java.util.InvalidPropertiesFormatException: org.xml.sax.SAXParseException:
    The processing instruction target matching "[xX][mM][lL]" is not allowed.
    at java.util.XMLUtils.load(Unknown Source)
    <?xml version="1.0"?>
    <Config>
         <Database>
              <Product>FX.O</Product>
              <URL>jdbc:oracle:thin:@0.0.0.0:ABC</URL>
         </Database>
    </Config>Am I missing anything?

    Thanks a lot for all yr help.I have got the basics of
    the DTD
    However,when I say thus
    <!DOCTYPE Config SYSTEM "c:/PartyConfig">
    ?xml version="1.0"?>
    <Config>
         <Database>
              <Product>FX</Product>
              <URL>jdbc:oracle:thin:@0.0.0.0:ABC</URL>
         </Database>
    </Config>I get the error
    Invalid system identifier: file:///c:/ParyConfig.dtd
    Please advise?for goodness sake, how many times do I have to tell you that you can't just expect the Properties class to accept arbitrary XML in any format? it reads XML in the format I linked to, and nothing else. Properties is not a general purpose XML binding. you can't do what you're trying to do with it. read this and in particular pay attention to the following
    The XML document must have the following DOCTYPE declaration:
    <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
    Furthermore, the document must satisfy the properties DTD described above.
    you are not reading my answers properly, are you? kindly do so

  • Exception occured while reading the license file.

    Java Exception: java.lang.RuntimeException
    Exception occurred while reading the license file.
    java.lang.RuntimeException: Exception occurred while reading the license file.
    at weblogic.security.utils.SSLSetup.getLicenseLevel(SSLSetup.java:143)
    at weblogic.security.utils.SSLSetup.getSSLDelegateInstance(SSLSetup.java:196)
    at weblogic.security.utils.SSLContextWrapper.<init>(SSLContextWrapper.java:27)
    at weblogic.security.utils.SSLContextWrapper.getInstance(SSLContextWrapper.java:22)
    at weblogic.security.utils.SSLSetup.getSSLContext(SSLSetup.java:380)
    at weblogic.security.SSL.SSLClientInfo.getSSLSocketFactory(SSLClientInfo.java:56)
    at weblogic.socket.ChannelSSLSocketFactory.getSocketFactory(ChannelSSLSocketFactory.java:125)
    at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:75)
    at weblogic.socket.AbstractMuxableSocket.createSocket(AbstractMuxableSocket.java:212)
    at weblogic.rjvm.t3.MuxableSocketT3.newSocketWithRetry(MuxableSocketT3.java:252)
    at weblogic.rjvm.t3.MuxableSocketT3.connect(MuxableSocketT3.java:421)
    at weblogic.rjvm.t3.MuxableSocketT3S.createConnection(MuxableSocketT3S.java:79)
    at weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1729)
    at weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1392)
    at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:445)
    at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:323)
    at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:263)
    at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:206)
    at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
    at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
    at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
    at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:342)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
    at weblogic.security.service.SecurityManager.runAs(Unknown Source)
    at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:337)
    at weblogic.jndi.Environment.getContext(Environment.java:307)
    at weblogic.jndi.Environment.getContext(Environment.java:277)
    at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
    at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
    at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
    at javax.naming.InitialContext.init(Unknown Source)
    at javax.naming.InitialContext.<init>(Unknown Source)
    I read in the community, that i need to set -Dbea.home="Directory", the Directory should refer to the path where license.bea file is kept on the client side. After setting this i need to pass the
    -Dbea.home="path" to the client jvm.
    Am using tomcat 6.0 server, and weblogic server 10.0.1.  I kept license.bea file in bin folder of the tomcat server.  But i dint understand how to set -Dbea.home="path" and how to pass it to the client jvm. Am stucked up hear. Can pls anyone help me,  to overcome this issue.

    Hi,
    Thank you for contacting us, the best community to place your question is at the OTN JAVA: https://home.java.net/?intcmp=JNET-javanet-sm
    Please let us know if you have any question about OPN CLOUD! Have a great day!

  • Exception occurred while reading the license file.

    Hi,
    I am having the Weblogic 9.2 server installed on 64 bit JVM on Solaris. I have some EJB's deployed over there. When I use to try to connect to these EJB's from Windows 32 bit OS I am getting the following error.
    Exception in thread "main" java.lang.RuntimeException: Exception occurred while reading the license file.
         at weblogic.security.utils.SSLSetup.getLicenseLevel(SSLSetup.java:143)
         at weblogic.security.utils.SSLSetup.getSSLDelegateInstance(SSLSetup.java:196)
         at weblogic.security.utils.SSLContextWrapper.<init>(SSLContextWrapper.java:27)
         at weblogic.security.utils.SSLContextWrapper.getInstance(SSLContextWrapper.java:22)
         at weblogic.security.utils.SSLSetup.getSSLContext(SSLSetup.java:380)
         at weblogic.security.SSL.SSLClientInfo.getSSLSocketFactory(SSLClientInfo.java:56)
         at weblogic.socket.ChannelSSLSocketFactory.getSocketFactory(ChannelSSLSocketFactory.java:125)
         at weblogic.socket.ChannelSSLSocketFactory.createSocket(ChannelSSLSocketFactory.java:75)
         at weblogic.socket.AbstractMuxableSocket.createSocket(AbstractMuxableSocket.java:207)
         at weblogic.rjvm.t3.MuxableSocketT3.newSocketWithRetry(MuxableSocketT3.java:252)
         at weblogic.rjvm.t3.MuxableSocketT3.connect(MuxableSocketT3.java:421)
         at weblogic.rjvm.t3.MuxableSocketT3S.createConnection(MuxableSocketT3S.java:79)
         at weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1749)
         at weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
         at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
         at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
         at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
         at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:350)
         at weblogic.jndi.Environment.getContext(Environment.java:307)
         at weblogic.jndi.Environment.getContext(Environment.java:277)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:247)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at Test.main(Test.java:26)
    But the same client works when I use it from same Solaris machine....
    Please help with this to me. This is argent for me.
    Thanks in advance ,Sachin

    Thanks for reply... I am using all dependent files at the client side...I had license.bea as well at the client side but it was not in the root directory so after I put it in the root directory that error was gone.....
    But I have issue with Certificates now... Actually I am using 32 bit jvm as client and the Weblogic is on 64 bit..
    The error I am facing now is......
    <Apr 17, 2009 2:13:41 PM GMT+05:30> <Warning> <Security> <BEA-090542> <Certificate chain received from ctsa-qa-s03 - 137.72.228.123 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.>
    Exception in thread "main" javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3s://ctsa-qa-s03:7009: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090542]Certificate chain received from ctsa-qa-s03 - 137.72.228.123 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.; No available router to destination]
         at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:49)
         at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:773)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:365)
         at weblogic.jndi.Environment.getContext(Environment.java:307)
         at weblogic.jndi.Environment.getContext(Environment.java:277)
         at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
         at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
         at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:247)
         at javax.naming.InitialContext.init(InitialContext.java:223)
         at javax.naming.InitialContext.<init>(InitialContext.java:197)
         at Test.main(Test.java:26)
    Caused by: java.net.ConnectException: t3s://ctsa-qa-s03:7009: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090542]Certificate chain received from ctsa-qa-s03 - 137.72.228.123 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.; No available router to destination
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:204)
         at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
         at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:350)
         ... 8 more
    Caused by: java.rmi.ConnectException: Destination unreachable; nested exception is:
         javax.net.ssl.SSLKeyException: [Security:090542]Certificate chain received from ctsa-qa-s03 - 137.72.228.123 was not trusted causing SSL handshake failure. Check the certificate chain to determine if it should be trusted or not. If it should be trusted, then update the client trusted CA configuration to trust the CA certificate that signed the peer certificate chain. If you are connecting to a WLS server that is using demo certificates (the default WLS server behavior), and you want this client to trust demo certificates, then specify -Dweblogic.security.TrustKeyStore=DemoTrust on the command line for this client.; No available router to destination
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:475)
         at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
         at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
         at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
         at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
         at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
         ... 10 more
    Please help me in this if you can

  • Upload very large file to SharePoint Online

    Hi,
    I tried uploading very large files to SharePoint online via the REST API using JAVA. Uploading files with a size of ~160 mb works well, but if the filesize is between 400mb and 1,6 gb I either receive a Socket Exception (Connection reset by peer) or an error
    of SharePoint telling me "the security validation for the page is invalid". 
    So first of all I want to ask you, how to work around the security validation error. As far as I understand the token, which I have added to my X-RequestDigest header of the http post request seems to be expired (by default it expires after 1800 seconds).
    Uploading such large files is time consuming, so I can't change the token in my header while uploading the file. How can I extend the expiration time of the token (if it is possible)? Could it help to send post requests with the same token
    continuously while uploading the file and therefore prevent the token from expiration? Is there any other strategy to upload such large files which need much more than 1800 seconds for upload?
    Additionally any thoughts on the socket exception? It happens quite sporadically. So sometimes it happens after uploading 100 mb, and the other time I have already uploaded 1 gb.
    Thanks in advance!

    Hi,
    Thanks for the reply, the reason I'm looking into this is so users can migrate there files to SharePoint online/Office 365. The max file limit is 2gb, so thought I would try and cover this.
    I've looked into that link before, and when I try and use those end points ie StartUpload I get the following response
    {"error":{"code":"-1, System.NotImplementedException","message":{"lang":"en-US","value":"The
    method or operation is not implemented."}}}
    I can only presume is that these endpoints have not been implemented yet, even though the documentation says they only available for Office 365 which is what  I am using. 
    Also, the other strange thing is that I can actually upload a file of 512mb to the server as I can see it in the web UI. The problem I am experiencing is get a response, it hangs on this line until the TimeOut is reached.
    WebResponse wresp = wreq.GetResponse();
    I've tried flushing and closing the stream, sendchunked but still no success

  • I/O exception while reading

    I got an error I/O exception while reading : E:\SampleCheckboxes2.html (The system can not find the file specified )
    What does it mean ??? what should I do ? I am very new in Java

    It means the file is not there. Check to make sure the file is in the specified location, if it is, check the spelling.

  • Very large file upload 2 GB with Adobe Flash

    Hello, anyone know how can I upload very large files with Adobe Flash or how to use SWFUpload?
    Thanks in Advance, All help will be very appreciated.

    1. yes
    2. I'm getting error message from php
    if( $_FILES['Filedata']['error'] == 0 ){ 
      if(move_uploaded_file( $_FILES['Filedata']['tmp_name'], $uploads_dir.$_FILES['Filedata']['name'] ) ){ 
      echo 'ok'; 
      exit(); 
      echo 'error';    // Overhere
      exit();

  • Unable to copy very large file to eSATA external HDD

    I am trying to copy a VMWare Fusion virtual machine, 57 GB, from my Macbook Pro's laptop hard drive to an external, eSATA hard drive, which is attached through an ExpressPort adapter. VMWare Fusion is not running and the external drive has lots of room. The disk utility finds no problems with either drive. I have excluded both the external disk and the folder on my laptop hard drive that contains my virtual machine from my Time Machihne backups. At about the 42 GB mark, an error message appears:
    The Finder cannot complete the operation because some data in "Windows1-Snapshot6.vmem" could not be read or written. (Error code -36)
    After I press OK to remove the dialog, the copy does not continue, and I cannot cancel the copy. I have to force-quit the Finder to make the copy dialog go away before I can attempt the copy again. I've tried rebooting between attempts, still no luck. I have tried a total of 4 times now, exact same result at the exact same place, 42 GB / 57 GB.
    Any ideas?

    Still no breakthrough from Apple. They're telling me to terminate the VMWare processes before attempting the copy, but had they actually read my description of the problem first, they would have known that I already tried this. Hopefully they'll continue to investigate.
    From a correspondence with Tim, a support representative at Apple:
    Hi Tim,
    Thank you for getting back to me, I got your message. Although it is true that at the time I ran the Capture Data program there were some VMWare-related processes running (PID's 105, 106, 107 and 108), this was not the case when the issue occurred earlier. After initially experiencing the problem, this possibility had occurred to me so I took the time to terminate all VMWare processes using the activity monitor before again attempting to copy the files, including the processes mentioned by your engineering department. I documented this in my posting to apple's forum as follows: (quote is from my post of Feb 19, 2008, 1:28pm, to the thread "Unable to copy very large file to eSATA external HDD", relevant section in >bold print<)
    Thanks for the suggestions. I have since tried this operation with 3 different drives through two different interface types. Two of the drives are identical - 3.5" 7200 RPM 1TB Western Digital WD10EACS (WD Caviar SE16) in external hard drive enclosures, and the other is a smaller USB2 100GB Western Digital WD1200U0170-001 external drive. I tried the two 1TB drives through eSATA - ExpressPort and also over USB2. I have tried the 100GB drive only over USB2 since that is the only interface on the drive. In all cases the result is the same. All 3 drives are formatted Mac OS Extended (Journaled).
    I know the files work on my laptop's hard drive. They are a VMWare virtual machine that works just fine when I use it every day. >Before attempting the copy, I shut down VMWare and terminated all VMWare processes using the Activity Monitor for good measure.< I have tried the copy operation both through the finder and through the Unix command prompt using the drive's mount point of /Volumes/jfinney-ext-3.
    Any more ideas?
    Furthermore, to prove that there were no file locks present on the affected files, I moved them to a different location on my laptop's HDD and renamed them, which would not have been possible if there had been interference from vmware-related processes. So, that's not it.
    Your suggested workaround, to compress the files before copying them to the external drive, may serve as a temporary workaround but it is not a solution. This VM will grow over time to the point where even the compressed version is larger than the 42GB maximum, and compressing and uncompressing the files will take me a lot of time for files of this size. Could you please continue to pursue this issue and identify the underlying cause?
    Thank you,
    - Jeremy

  • Web Service to transmit a file (a very large file)

    Greetings,
    Is is possible to use a web service to transmit a file, sometimes a very large file? If so, what are the security implications of doing such a thing?
    Thanks,
    Charles

    Quick answer: It depends... Did you try using Google to look-up examples of doing this? Here is a url to try: http://www.google.com/#sclient=psy&hl=en&source=hp&q=web+service+transfer+file&pbx=1&oq=web+service+transfer+file&aq=f&aqi=g1g-j3g-b1&aql=&gs_sm=e&gs_upl=1984731l1997518l0l1997874l25l19l0l1l1l0l304l3879l0.7.10.1l18l0&bav=on.2,or.r_gc.r_pw.&fp=7caaed5eb0414d97&biw=1280&bih=871
    Thank you,
    Tony Miller
    Webster, TX
    Never Surrender Dreams!
    JMS
    If this question is answered, please mark the thread as closed and assign points where earned..

  • Exporting to iTunes results in a very large file

    I have been making music on Garageband and when I export it to itunes, it becomes a very large file. This makes disables me from being able to post my music on my website or email it. What can I do to make the file smaller?
    imac G5   Mac OS X (10.4.5)   1.9 GHz PowerPC G5

    You should convert your file to an MP3 in iTunes:
    - Make sure that in the import options of iTunes, MP3 is selected.
    - Mark your song in iTunes and select "convert to MP3" in the menu.
    - This creates a new file in iTunes with the same title. If you don't know which is which, apple-I will tell you if it's an AIFF or an MP3, and the file size.

  • I inadvertently created a very large file. Not needing it I sent it to the Trash Can. However, the deletion process never completes. Any suggestions as to how to delete it ?

    I inadvertently created a very large file on my hard drive. Not needing it I sent it to the Trash Can. However, the deletion process never completes. Any suggestions as to how to delete it ? I re-installed the OS but the file was still there taking up needed space.

    Command-click all of the files you want to delete and don't actually move them to the Trash until you've gone through the entire folder.
    (63341)

  • How can I get to read a large file and not break it up into bits

    Hi,
    How do I read a large file and not get the file cut into bits so each has its own beginning and ending.
    like
    1.aaa
    2.aaa
    3.aaa
    4....
    10.bbb
    11.bbb
    12.bbb
    13.bbb
    if the file was read on the line 11 and I wanted to read at 3 and then read again at 10.
    how do I specify the byte in the file of the large file since the read function has a read(byteb[],index,bytes to read);
    And it will only index in the array of bytes itself.
    Thanks
    San Htat

    tjacobs01 wrote:
    Peter__Lawrey wrote:
    Try RandomAccessFile. Not only do I hate RandomAccessFiles because of their inefficiency and limited use in today's computing world, The one dominated by small devices with SSD? Or the one dominated by large database servers and b-trees?
    I would also like to hate on the name 'RandomAccessFile' almost always, there's nothing 'random' about the access. I tend to think of the tens of thousands of databases users were found to have created in local drives in one previous employer's audit. Where's the company's mission-critical software? It's in some random Access file.
    Couldn't someone have come up with a better name, like NonlinearAccessFile? I guess the same goes for RAM to...Non-linear would imply access times other than O(N), but typically not constant, whereas RAM is nominally O(1), except it is highly optimised for consecutive access, as are spinning disk files, except RAM is fast in either direction.
    [one of these things is not like the other|http://www.tbray.org/ongoing/When/200x/2008/11/20/2008-Disk-Performance#p-11] silicon disks are much better at random access than rust disks and [Machine architecture|http://video.google.com/videoplay?docid=-4714369049736584770] at about 1:40 - RAM is much worse at random access than sequential.

  • Please advice me on reading a large file

    Hi,
    I need and advice on reading a large file using JSP. I need to read a text file which contains around 2 millions records (lines). Also some times I need to find a specific line from that.
    I have to make a small web application to generate reports on the Text files after parsing those into a database (Oracle or MySQL). I have already a small similar application which based on PHP / MySQL. But still time consumption issue is there while parsing the text file into Database.
    How will it be improve if I go for Tomcat / Oracle / Java Web Component Development?
    Please advice me.
    Thanks

    Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

  • Sorting very large file

    Hi,
    I've a very large file (1.3G, more than 10 million records) that I need to sort, what is the best possible way to do that, without using a database?
    Thanks,
    Gary

    Just a suggestion:
    If you create a sorted red/black tree of record numbers (i.e. offsets in the data file), you could get an equivalent sort as follows (probably without a huge amount of memory...):
    Here's the strategy:
    Create a comparitor that compares two Long objects. It performs a seek in the data file and retrieves the records pointed to by those two Longs and compares them using whatever record comparitor you were going to use.
    Create a sorted list based on that comparitor (a tree based list will probably be best)
    Run a for loop going from 0 to the total records in the data file, create Long objects for each value, and insert that value into the list.
    When you are done, the list will contain the record numbers in sorted order.
    Create an iterator for the list
    For each element in the list, grab the corresponding data file record from the source file and write it to the destination file.
    voila
    Some optimizations may be possible (and highly desirable, actually):
    When you read records from the file, place them into an intermediate cache (linked list?) of limited length (say 1000 records). When a record is requested, scan the cache for that record ID - if found, move it to the top of the list, then return the associated data. If it is not found, read from the file, then add to the top of the list. If the list has more than X records, remove the last item from the list.
    The upper pieces of the sort tree are going to get hammered, so having a cache like this is a good idea.
    Another possibility: Instead of inserting the records in order (i.e. 0 through X), it may be desirable to insert them in pseudo-random order - I seem to remember that there was a really nice algorithm for doing this (without repeating any records) in an issue of Dr. Dobbs from awhile back. You need to get a string of random numbers between 0 and X without repeats (like dealing from a card deck).
    If your data is already in relatively sorted order, the cache idea will not provide significant bennefit without randomizing the inputs.
    One final note: Try the algorithm without the cache first and see how bad it is. The OSes disk caching may provide sufficient performance. If you encapsulate the data file (i.e. an object that has a getRecordBytes(int recNumber) method), then it will be easy to implement the cache later if it is needed.
    I hope this provides some food for thought!
    - K

Maybe you are looking for

  • How Can I Get the FULL Installer for Flash Player 11.1.102.63?

    When I try the "Download now" button at http://get.adobe.com/flashplayer/, I only get a stub installer that then requires downloading the remainder of the installer.  I maintain both my own PC and my wife's PC.  I want to install Flash Player 11.1.10

  • Dates are not flowing properly into bw

    Hi Friends, We have some serious problem. we have few reports on Sales and the data is extracted from r/3 into bw on daily basis. some times the reports working fine other times either the production date or exp date is not flowing into BW or it is g

  • How to use standard text feature SO10 , dynamic text ?

    hi i have to pass an email containing some information, information contains employee name and number , that I have to place at runtime in the message which is created in the SO10. How to do this ? thanks,

  • How to use awr and addm reports

    Hi, to use awr and addm reports does we should have separate license or oracle server software license is enough. which tool do we use to read this awr and addm reports in real time. please give your valuable suggestions. thank you!

  • Video Clip - computer freeze

    What can I do to prevent my computer from freezing up when I begin to run a video clip in iphoto