Please advice me on reading a large file

Hi,
I need and advice on reading a large file using JSP. I need to read a text file which contains around 2 millions records (lines). Also some times I need to find a specific line from that.
I have to make a small web application to generate reports on the Text files after parsing those into a database (Oracle or MySQL). I have already a small similar application which based on PHP / MySQL. But still time consumption issue is there while parsing the text file into Database.
How will it be improve if I go for Tomcat / Oracle / Java Web Component Development?
Please advice me.
Thanks

Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

Similar Messages

  • How can I get to read a large file and not break it up into bits

    Hi,
    How do I read a large file and not get the file cut into bits so each has its own beginning and ending.
    like
    1.aaa
    2.aaa
    3.aaa
    4....
    10.bbb
    11.bbb
    12.bbb
    13.bbb
    if the file was read on the line 11 and I wanted to read at 3 and then read again at 10.
    how do I specify the byte in the file of the large file since the read function has a read(byteb[],index,bytes to read);
    And it will only index in the array of bytes itself.
    Thanks
    San Htat

    tjacobs01 wrote:
    Peter__Lawrey wrote:
    Try RandomAccessFile. Not only do I hate RandomAccessFiles because of their inefficiency and limited use in today's computing world, The one dominated by small devices with SSD? Or the one dominated by large database servers and b-trees?
    I would also like to hate on the name 'RandomAccessFile' almost always, there's nothing 'random' about the access. I tend to think of the tens of thousands of databases users were found to have created in local drives in one previous employer's audit. Where's the company's mission-critical software? It's in some random Access file.
    Couldn't someone have come up with a better name, like NonlinearAccessFile? I guess the same goes for RAM to...Non-linear would imply access times other than O(N), but typically not constant, whereas RAM is nominally O(1), except it is highly optimised for consecutive access, as are spinning disk files, except RAM is fast in either direction.
    [one of these things is not like the other|http://www.tbray.org/ongoing/When/200x/2008/11/20/2008-Disk-Performance#p-11] silicon disks are much better at random access than rust disks and [Machine architecture|http://video.google.com/videoplay?docid=-4714369049736584770] at about 1:40 - RAM is much worse at random access than sequential.

  • Reader can't read/open large files created in ArcGIS ??

    Hi there
    We have a problem with files (GEO PDF's) created primarily from ArcGIS (ESRI).
    Reader version XI or DC can't open/read them (blank screen, but all functions working). It can be files around 60 MB or 1.5 GB and above 300 DPI.
    The weird thing is that if we use FOXIT ALL of them will open/show so it can't be the file that is corrupt, but something in Reader.....either a setting, limitation or ??
    Anyone ??????

    Hi,
    We would like to take a look at the large files to investigate the cause of the problem.
    Please note that the Adobe forums do not accept email attachments.
    Would you share the links to the files?
    How to share a document
    If you are not comfortable sharing the links in this public user forum, please send me a private forum message with the links.
    Thank you.

  • Reading a large file?  I created an Applescript to read an XML document...

    I created an Applescript to read an XML document that had been exported from Final Cut Pro but it has trouble reading anything larger than a 1mb. Even 500kb takes a long time.
    My code is most likely very ineffecient, but I have little experience with Applescript. The script reads an XML document one line at a time then breaks down each line into it's componenents and literally reads each character looking for "<name>" then it checks to see if "<reel>" came before it. It then records the name in between "<name>" and "</name>." Then I get my list of tapes that are in the Final Cut Pro project. Does anyone have any advice on how to improve this code?
    property type_list : {"TEXT", "XML"}
    property extension_list : {"TXT", "XML"}
    on open these_items
    tell application "Finder"
    set my_file to these_items as string
    set file_ref to (open for access my_file) read
    close access file my_file
    set AppleScript's text item delimiters to ASCII character 10
    set new_list to every text item of file_ref
    set lengthofarray to length of new_list
    set h to 1
    set shotcount to 0
    set finalShots to {"You have ", "", " shots in your project."}
    set finalListoftapes to {"Your list of tapes are:", ""}
    repeat lengthofarray times
    set i to 1
    set x to 1
    set switch to 0
    set z to {""}
    set reelName to {"", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""}
    set letter_list to item h of new_list
    set AppleScript's text item delimiters to ""
    set final_list to every text item of letter_list
    set lengthofletter_list to length of final_list
    repeat lengthofletter_list times
    if item i of final_list is "<"a" then
    if item (i + 3) of final_list is "m" then
    set letter_list2 to item (h - 1) of new_list
    set final_list2 to every text item of letter_list2
    set lengthofletter_list2 to length of final_list2
    set j to 1
    repeat lengthofletter_list2 times
    if item j of final_list2 is "<" then
    if item (j + 1) of final_list2 is "r" then
    if item (j + 2) of final_list2 is "e" then
    set x to (i + 6)
    set y to 1
    repeat while item 1 of z is not "<"
    set item y of reelName to item x of final_list
    set x to (x + 1)
    set y to (y + 1)
    set item 1 of z to item x of final_list as string
    end repeat
    if item 1 of reelName is not "" then
    set displayText to reelName as string
    set lengthofListoftapes to length of finalListoftapes
    set shotcount to (shotcount + 1)
    set k to 1
    repeat lengthofListoftapes times
    if item k of finalListoftapes is equal to displayText then
    set switch to 1
    end if
    set k to (k + 1)
    end repeat
    if switch is 0 then
    set finalListoftapes to (finalListoftapes & displayText)
    set check to finalListoftapes as string
    end if
    set switch to 0
    end if
    end if
    end if
    end if
    set j to (j + 1)
    end repeat
    end if
    end if
    end if
    end if
    set i to (i + 1)
    end repeat
    set h to (h + 1)
    end repeat
    set item 2 of finalShots to shotcount
    set finalShots to finalShots as string
    set AppleScript's text item delimiters to ASCII character 10
    set finalListoftapes to finalListoftapes as string
    set finalListoftapes to finalShots & (ASCII character 10) & (ASCII character 10) & finalListoftapes & (ASCII character 10) & (ASCII character 10) & "This list of tapes has been saved on your desktop."
    set path_list to these_items as string
    set filepath to path_list & "_listoffiles.txt"
    set outputFile to (open for access file filepath with write permission)
    write finalListoftapes to outputFile
    close access outputFile
    display dialog finalListoftapes
    end tell
    end open

    Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

  • Please help me in reading the ascii file in encoded format

    hi ,
    iam trying to read the ascii file and i need to encode the text file, can u suggest me what arre the different methods are availble for me to encode the text file, please suggest me which is the effiecient method tto use?

    This question has been answered before, please search the forums in future.
    You could do something like this, it probably not the most efficient though.
    If you don't need to do it in code then the native2ascii does this conversion see: http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/native2ascii.html
            public static void convert(String vsFile) throws IOException
              File f = new File(vsFile);
              FileReader fr = new FileReader(f);
              char[] buf;
              int bytesRead = 0;
              int startAt = 0;
              String content = "";
              do
                   buf = new char[512];
                   bytesRead = fr.read(buf, 0, 512);
                   startAt += bytesRead;
                   String tmp = new String(buf);
                   content += tmp;
              while (bytesRead == 512);
              FileOutputStream fout = new FileOutputStream(f);
              OutputStreamWriter out = new OutputStreamWriter(fout, "UTF-8");
              out.write(content.trim());
              out.flush();
              out.close();
         }

  • Exception while Read very large file 300 MB

    Hi
    I want to read a file which has size of more than 300 MB.
    While i m executing my code, i m getting
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    Below is my code
    1. FileChannel fc = new FileInputStream(fFile).getChannel();
    2.CharBuffer chrBuff = Charset.forName("8859_1").newDecoder().decode(fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size()));
    3.fc.close();Im getting the exception in the line 2.
    Even though i increased the Heap space up to 900 Mb, i m getting this error.
    (FYI)
    I executed in command prompt like below,
    java -Xms128m -Xmx1024m readLargeFile
    Kindly give a solution for this. is there any other best way to read large text file ??
    I m waiting for your reply..
    Thanks in advance

    Thanks for your reply.
    My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
    Then i need to write that searched area of text to a new file and delete that portion from the original file.
    The above process i will do more times.
    So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.
    Above is my task, then now suggest me some efficient ways.
    Note that my file size will be more and using JAVA only i've to do these things.
    Thanks in Advance...

  • Reading A Large File At Launch

    Hi all,
    I've got my app working on the ad hoc iPhones of my testers.
    At present, it loads a 6MB data file (a Property List) using the NSDictionary initWithContentsOfFile method. Of course, this is nearly instantaneous on the simulator, but my beta-testers report that the launch time of my app is 10 to 15 seconds because of this.
    If I could go ahead and present my User Interface to the user while this file was being read by another thread or NSOperation, with the only restriction being that a certain method in my code could not be called until the file was completely read in, then this would improve the perceived responsiveness of my app.
    I don't want to "read as needed", as I want the app to have the whole file in memory once it is ready to use.
    Can someone suggest the absolute easiest way to let the UI go ahead and appear while an XML plist is being read into an NSDictionary in a parallel operation/task/thread?
    Right now the -initWithContentsOfFile is in the -applicationDidFinishLaunching delegate method and I would somehow like to spin that reading of the file off and let the code continue on to load the interface and allow the user to start typing in a text box.
    The only "coordination" I need to do is make sure the file has finished being read before executing a method that looks up a key in the NSDictionary.
    What's your opinion on the easiest way to do this? NSOperation?
    I would truly appreciate any advice on this. Thanks.

    I'm not sure about Cryptix but the JDK includes APIs for encrypting/decyrpting streams.
    Jim

  • What file type should i use for reading a large file of data?

    I want to store data in a file and access them later. The data file could reach 500 to 1000 MBytes, as a binary file.
    I may have memory problems.
    I do not need the whole data in memory, only a few data each time, for my calculations.
    If i use a binary file, i can not read only a few data. I have to load the whole file and then read the data i need.
    Is this correct?
    If i use another type of file, can i read only a few bytes without loading the whole file?
    Maybe a TDMS file?
    Solved!
    Go to Solution.

    I would probably use a TDMS file for this since it could also be read into excel if it were small enough - just more flexiblility. But you can also do it using binary types. You don't have to read the entire file when using binary files. See below.

  • I need help on reading a large file, thanks.

    I try to read a txt file into an array, just like what I ususally do. But this time the file is too big (300k rows, 100 columns seperated by single space), I got the message "java.lang.OutOfMemoryError".
    I need to do some calculation for every possible row pair. Can someone help me out of this? Thanks a lot.

    Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

  • JEditorPane reading in large files

    Hello,
    I am currently using JEditorPane.read() to read a rather big ASCII text file. It works for the most part except:
    1. The java runtime dies due to out of memory exception for any file that is over around 6 megabytes in size.
    2. Any file whose size is around 5.5 - 6 megabytes tables around a minute to load. I would like to cut down on the loading time.
    Does anyone have any suggestions? Is there an alternative besides using JEditorPane as a view into a large log file? I am using it as view only and not editing any data with it.
    Thanks
    Jim

    my collegue has a similar problem.
    have you fixed this yet ??
    do you use fileInputStream or as suggested in various
    replies i've seen use the BufferedFileInPutStream.
    hope this helps,,,, let me know your status
    thanks
    Rob.

  • Please advice about fast reading/writing

    Hi All,
    I am building a custom system for Photoshop CS6 and Lightroom 4, I know this is not a PS forum but I think the hardware experts are here.
    The components I have already selected are:
    i7-3930k
    ASUS P9X79 PRO
    64GB RAM (GSKILL)
    WD Caviar Black 1TB
    Since I have enough RAM the scratch disk is not something to worry about, after reading this article about SSDs for PS:
    http://helpx.adobe.com/photoshop/kb/optimize-performance-photoshop-cs4-cs5.html#main_Solid _state_disks
    I think a SSD is useful only when starting the applications (having them in the SSD), which is not important for me (I do it twice or three times per day). What I would like to do (and is the matter of this post) is to open and save my large pictures (700MB to 1+ GB) as kickly as posibble, is the Caviar Black a good choice or should I consider a SSD? What about a WD velociraptor? Also, I would appreciate advice about other options.
    The last custom build I did was probably 8 years ago so I am not up to date with the hardware technology, I have learned a lot reading this great forum :-).
    Thank you,
    William Gonzales

    William,
    I use Photoshop CS5 and Bridge a lot and care particularly about how fast the PC can read since this dramatically impacts how fast you can process and rank lots of shots.
    Bottom line...
    Suggest in addition to whatever your boot drive is you configure a 2xRAID 0 7200rpm drive setup for your data. You have an extremely capable CPU and lots of RAM, so your disk speed is definitely a bottleneck.
    Another alternative, although more expensive and quite a bit smaller would be one of the latest, fastest SSDs that can finally deliver on the write side. (i.e. Samsung 840 Pro and its competitors). This would be a secondary drive to whatever you have for your boot drive.
    Another suggestion, is make sure you have a video card with good 2D speed. That is get one that can move lots of data; this spec. (Memory Bandwidth, often in GB/sec) is often buried since gaming and 3D features (cores, shading, memory size, etc.) are most important for most users. My video card, a GTX 480 has a memory bandwidth of 177.4 GB/sec, but probably any card with at least 80 GB/sec will probably work very well with your apps.
    Regards,
    Jim

  • How to read a large file(size around 100 kbytes) into a byte array in JAVA

    aa

    But I cannot use array subscript as a long value.what do you mean?
    are you rying the following:
    long aSize = 100000;
    byte[] array = new byte[aSize];
    if this is the case, then don't do it, for your 100k file int is far more than enough.
    but in case you jsut get error while:
    byte b = array[100000];
    then it's probably because you'r array is of size 100000, that means you have bytes from 0 to 99999 in there, and 100000 is one passed the end of the array.... jus in case make your array size to be 1234567 (far more than needed for your file) and see if you still get that exception, if not, then your array was too small. but if it fixes your prob, then don't leave it as it is, try to work out the right size for array and use that....
    i wouldn't recommend using static array size, since that file might some day be bigger than you expected, and then you'd have to recompile your program.
    HTH

  • When saving an downloaded pdf-fil, the saved file is corrupt (empty?) and can´t be opened. Happens all the time nowadays, never before. Its a mess. Please advice.

    When saving an downloaded pdf-fil, the saved file is corrupt (empty?) and can´t be opened. Happens all the time nowadays, never before. Its a mess. Please advice.

    Hi Melbus_SE,
    Are you downloading files from your Acrobat.com cloud storage? If so, what browser are you using, and are you on Mac OS or Windows? Has something changed on your system since you were able to download/open PDF files successfully? What version of Acrobat or Reader are you using to open these files?
    Sorry to ask so many questions, but your answers will help us get to the bottom of this issue.
    I look forward to hearing back from you.
    Best,
    Sara

  • UTL_FILE.get_line won't read large files ?

    I am trying to read a large fixed length flat file. If I cut the file down to really small it will read it but it reads it as a single line. If I try to read a larger file > 32k I get a READ_ERROR. I am pretty sure it has to do with the end of line marker but I saw nothing about that in the UTL_FILE documentation. This is on UNIX, new line character after each record in the file. Standard unix flat file.
    Any ideas on what to do?
    Thanks in advance
    Matt
    [email protected]
    my code:
    BEGIN
    BEGIN
    std_file := UTL_FILE.FOPEN('&4','&1','r',32767);
    EXCEPTION
    WHEN UTL_FILE.INVALID_PATH THEN
    RAISE_APPLICATION_ERROR(-20011,'Invalid Path for STD file, &4/&1');
    WHEN OTHERS THEN
    RAISE_APPLICATION_ERROR(-20014,'Other Error trying to open STD file, &4/&1');
    END;
    IF UTL_FILE.is_open(std_file) = FALSE THEN
    RAISE_APPLICATION_ERROR(-20015,'Could not open STD file, &4/&1');
    END IF;
    -- READ STD FILE HEADER
    BEGIN
    UTL_FILE.get_line(std_file,hdr_text);
    EXCEPTION
    WHEN UTL_FILE.INVALID_FILEHANDLE THEN
    RAISE_APPLICATION_ERROR(-20017,'STD read file handle not valid');
    WHEN UTL_FILE.INVALID_OPERATION THEN
    RAISE_APPLICATION_ERROR(-20018,'STD read invalid operation error');
    WHEN UTL_FILE.READ_ERROR THEN
    RAISE_APPLICATION_ERROR(-20019,'STD read error');
    WHEN NO_DATA_FOUND THEN
    RAISE_APPLICATION_ERROR(-20020,'STD read no data found');
    WHEN VALUE_ERROR THEN
    RAISE_APPLICATION_ERROR(-20021,'STD read value error');
    END;
    -- PROCESS TRANSACTIONS
    LOOP
    BEGIN
    tran_text := NULL;
    UTL_FILE.get_line(std_file,tran_text);
    EXCEPTION
    WHEN no_data_found THEN EXIT; -- EOF
    WHEN value_error THEN
    RAISE_APPLICATION_ERROR(-20010,'STD record too long.');
    END;
    std_rowcount := std_rowcount + 1;
    END LOOP;
    UTL_FILE.FCLOSE(std_file);
    EXCEPTION
    WHEN NO_DATA_FOUND THEN
    RAISE_APPLICATION_ERROR(-20001,'No Data Found.');
    WHEN UTL_FILE.INVALID_PATH THEN
    RAISE_APPLICATION_ERROR(-20002,'Invalid Path ');
    WHEN UTL_FILE.INVALID_MODE THEN
    RAISE_APPLICATION_ERROR(-20003,'Invalid Mode ');
    WHEN UTL_FILE.INVALID_OPERATION THEN
    RAISE_APPLICATION_ERROR(-20004,'Invalid Operation ');
    END;
    null

    We are still hung up on this. I tried implementing the code from STEVE'S XML book but still haven't resovled it.
    The clob is being created via XSU see below. The new char[8192] appeasr to force the output file to 8K
    with trailing characters on small clobs but adds a carraige return each 8K on larger ones.
    As usual any input is appreciated from all. Doese anyone know of a good JAVA forum like this one?
    Thanks
    PROCEDURE BuildXml(v_return OUT INTEGER, v_message OUT VARCHAR2,string_in VARCHAR2,xml_CLOB OUT NOCOPY CLOB) IS
    queryCtx DBMS_XMLquery.ctxType;
    Buffer RAW(1024);
    Amount BINARY_INTEGER := 1024;
    Position INTEGER := 1;
    sql_string     VARCHAR2(2000) := string_in;
    BEGIN
    v_return := 1;
    v_message := 'BuildXml completed succesfully.';
    queryCtx := DBMS_XMLQuery.newContext(sql_string);
    xml_CLOB := DBMS_XMLQuery.getXML(queryCtx);
    DBMS_XMLQuery.closeContext(queryCtx);
    EXCEPTION WHEN OTHERS THEN
    v_return := 0;
    v_message := 'BuildXml failed - '||SQLERRM;
    END BuildXml;
    create or replace and compile java source named sjs.write_CLOB as
    import java.io.*;
    import java.sql.*;
    import java.math.*;
    import oracle.sql.*;
    import oracle.jdbc.driver.*;
    public class write_CLOB extends Object
    public static void pass_str_array(oracle.sql.CLOB p_in,java.lang.String f_in)
    throws java.sql.SQLException, IOException
    File target = new File(f_in);
    FileWriter fw = new FileWriter(target);
    BufferedWriter out = new BufferedWriter(fw);
    Reader is = p_in.getCharacterStream();
    char buffer[] = new char[8192];
    int length;
    while( (length=is.read(buffer)) != -1) {
    out.write(buffer);
    is.close();
    fw.close();
    /

  • A usefull way to read a long file

    Hi,
    What's the best and usefullest way to read a large file?
    I do use a LineNumberReader but i think it reads a little slow...
    To open the same file in an text editor reacts in less a millisecond while my LineNumberReader nees more time to read the entire file...
    Any suggestions?
    Olek

    re,
    how can i determine how a file contains "real" text or only sensles characters?
    The file i readed seems to be a binary file in your meaning.
    here is the method i use
    // Threader.FileReadThreader is a instance of SwingWorker
    public static String readPlainTextFile(File f, Threader.FileReadThreader fileReader) throws IOException {
            final StringBuilder strBuild = new StringBuilder();
            final DataInputStream dis = new DataInputStream(new BufferedInputStream(new FileInputStream(f)));
            final int readByte = BUFFER;
            final int fileLength = (int) f.length();
            float step = fileLength / readByte;
            byte[] buffer = new byte[fileLength];
            int off = 0;
            if (fileReader != null) {
                float percent = 0F;
                while (off < fileLength - readByte) {
                    dis.readFully(buffer, off, readByte);
                    String tempSequence = new String(Arrays.copyOfRange(buffer, off, off + readByte - 1));
                    strBuild.append(tempSequence);
                    off += readByte;
                    percent += step;
                    fileReader.doPublish(new Object[]{Math.round(percent), tempSequence});
                // read the remaining bytes
                int last = fileLength - off;
                dis.readFully(buffer, off, last - 1);
                String tempSequence = new String(Arrays.copyOfRange(buffer, off, off + last - 1));
                strBuild.append(tempSequence);
                percent += step;
                fileReader.doPublish(new Object[]{Math.round(percent), tempSequence});
                return strBuild.toString();
            } else {
                dis.readFully(buffer);
                return new String(buffer);
        }As you can see i modified your code.
    Hm i got the error why the JTextArea do only display
    one large line of characters...
    i forgot to set line wrap ...
    but now the app freezes reading the large binary file ...
    with a smaller file all works ok.
    Any ideas?
    regards.
    Olek

Maybe you are looking for

  • I keep getting a server error and my emails are not deleting and showing up multiple times

    I am getting emails multiple times and they aren't deleting.  They keep coming back.  I then get a server errer that mail.me may be a false server pretending to be mail.me.  Don't know how to fix this.

  • Sony Bravia high pitched ringing

    I just bought a brand new Sony Bravia 32' model KDL32EX500 yesterday and everything appeared to be in working order except for a very high pitched ringing sound that comes from the television. I figured this was normal as it is powered on, but it als

  • Ipad 3 - warranty worldwide?

    Hello everybody, I'm going to New York City this May (2012) and would like to buy the new ipad 3 there. Due to the low exchange rates, it would be a lot cheaper than here in Germany. The last thing I'm worrying about is the warranty. Unfortunately, t

  • Steps for Getting Repository Reports to Work in SCM 9024

    Have installed Des9i/SCM 9024. With this install, repository reports are not installed without the base release of O9iDS R2. What are the exact steps for getting the Designer Repository Reports to work via the RON in SCM 9024? Basically, I would like

  • Re-read in values in measurement file when file is updated

    Hi, I'm trying to control a power supply with labview and set the voltage. I've currently got it working reading in a measurement file and setting it to the value within the file. I'm now having the issue that I would like to update this value when t