Reading A Large File At Launch

Hi all,
I've got my app working on the ad hoc iPhones of my testers.
At present, it loads a 6MB data file (a Property List) using the NSDictionary initWithContentsOfFile method. Of course, this is nearly instantaneous on the simulator, but my beta-testers report that the launch time of my app is 10 to 15 seconds because of this.
If I could go ahead and present my User Interface to the user while this file was being read by another thread or NSOperation, with the only restriction being that a certain method in my code could not be called until the file was completely read in, then this would improve the perceived responsiveness of my app.
I don't want to "read as needed", as I want the app to have the whole file in memory once it is ready to use.
Can someone suggest the absolute easiest way to let the UI go ahead and appear while an XML plist is being read into an NSDictionary in a parallel operation/task/thread?
Right now the -initWithContentsOfFile is in the -applicationDidFinishLaunching delegate method and I would somehow like to spin that reading of the file off and let the code continue on to load the interface and allow the user to start typing in a text box.
The only "coordination" I need to do is make sure the file has finished being read before executing a method that looks up a key in the NSDictionary.
What's your opinion on the easiest way to do this? NSOperation?
I would truly appreciate any advice on this. Thanks.

I'm not sure about Cryptix but the JDK includes APIs for encrypting/decyrpting streams.
Jim

Similar Messages

How can I get to read a large file and not break it up into bits

Hi,
How do I read a large file and not get the file cut into bits so each has its own beginning and ending.
like
1.aaa
2.aaa
3.aaa
4....
10.bbb
11.bbb
12.bbb
13.bbb
if the file was read on the line 11 and I wanted to read at 3 and then read again at 10.
how do I specify the byte in the file of the large file since the read function has a read(byteb[],index,bytes to read);
And it will only index in the array of bytes itself.
Thanks
San Htat

tjacobs01 wrote:
Peter__Lawrey wrote:
Try RandomAccessFile. Not only do I hate RandomAccessFiles because of their inefficiency and limited use in today's computing world, The one dominated by small devices with SSD? Or the one dominated by large database servers and b-trees?
I would also like to hate on the name 'RandomAccessFile' almost always, there's nothing 'random' about the access. I tend to think of the tens of thousands of databases users were found to have created in local drives in one previous employer's audit. Where's the company's mission-critical software? It's in some random Access file.
Couldn't someone have come up with a better name, like NonlinearAccessFile? I guess the same goes for RAM to...Non-linear would imply access times other than O(N), but typically not constant, whereas RAM is nominally O(1), except it is highly optimised for consecutive access, as are spinning disk files, except RAM is fast in either direction.
[one of these things is not like the other|http://www.tbray.org/ongoing/When/200x/2008/11/20/2008-Disk-Performance#p-11] silicon disks are much better at random access than rust disks and [Machine architecture|http://video.google.com/videoplay?docid=-4714369049736584770] at about 1:40 - RAM is much worse at random access than sequential.

Please advice me on reading a large file

Hi,
I need and advice on reading a large file using JSP. I need to read a text file which contains around 2 millions records (lines). Also some times I need to find a specific line from that.
I have to make a small web application to generate reports on the Text files after parsing those into a database (Oracle or MySQL). I have already a small similar application which based on PHP / MySQL. But still time consumption issue is there while parsing the text file into Database.
How will it be improve if I go for Tomcat / Oracle / Java Web Component Development?
Please advice me.
Thanks

Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

Reader can't read/open large files created in ArcGIS ??

Hi there
We have a problem with files (GEO PDF's) created primarily from ArcGIS (ESRI).
Reader version XI or DC can't open/read them (blank screen, but all functions working). It can be files around 60 MB or 1.5 GB and above 300 DPI.
The weird thing is that if we use FOXIT ALL of them will open/show so it can't be the file that is corrupt, but something in Reader.....either a setting, limitation or ??
Anyone ??????

Hi,
We would like to take a look at the large files to investigate the cause of the problem.
Please note that the Adobe forums do not accept email attachments.
Would you share the links to the files?
How to share a document
If you are not comfortable sharing the links in this public user forum, please send me a private forum message with the links.
Thank you.

Reading a large file? I created an Applescript to read an XML document...

I created an Applescript to read an XML document that had been exported from Final Cut Pro but it has trouble reading anything larger than a 1mb. Even 500kb takes a long time.
My code is most likely very ineffecient, but I have little experience with Applescript. The script reads an XML document one line at a time then breaks down each line into it's componenents and literally reads each character looking for "<name>" then it checks to see if "<reel>" came before it. It then records the name in between "<name>" and "</name>." Then I get my list of tapes that are in the Final Cut Pro project. Does anyone have any advice on how to improve this code?
property type_list : {"TEXT", "XML"}
property extension_list : {"TXT", "XML"}
on open these_items
tell application "Finder"
set my_file to these_items as string
set file_ref to (open for access my_file) read
close access file my_file
set AppleScript's text item delimiters to ASCII character 10
set new_list to every text item of file_ref
set lengthofarray to length of new_list
set h to 1
set shotcount to 0
set finalShots to {"You have ", "", " shots in your project."}
set finalListoftapes to {"Your list of tapes are:", ""}
repeat lengthofarray times
set i to 1
set x to 1
set switch to 0
set z to {""}
set reelName to {"", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""}
set letter_list to item h of new_list
set AppleScript's text item delimiters to ""
set final_list to every text item of letter_list
set lengthofletter_list to length of final_list
repeat lengthofletter_list times
if item i of final_list is "<"a" then
if item (i + 3) of final_list is "m" then
set letter_list2 to item (h - 1) of new_list
set final_list2 to every text item of letter_list2
set lengthofletter_list2 to length of final_list2
set j to 1
repeat lengthofletter_list2 times
if item j of final_list2 is "<" then
if item (j + 1) of final_list2 is "r" then
if item (j + 2) of final_list2 is "e" then
set x to (i + 6)
set y to 1
repeat while item 1 of z is not "<"
set item y of reelName to item x of final_list
set x to (x + 1)
set y to (y + 1)
set item 1 of z to item x of final_list as string
end repeat
if item 1 of reelName is not "" then
set displayText to reelName as string
set lengthofListoftapes to length of finalListoftapes
set shotcount to (shotcount + 1)
set k to 1
repeat lengthofListoftapes times
if item k of finalListoftapes is equal to displayText then
set switch to 1
end if
set k to (k + 1)
end repeat
if switch is 0 then
set finalListoftapes to (finalListoftapes & displayText)
set check to finalListoftapes as string
end if
set switch to 0
end if
end if
end if
end if
set j to (j + 1)
end repeat
end if
end if
end if
end if
set i to (i + 1)
end repeat
set h to (h + 1)
end repeat
set item 2 of finalShots to shotcount
set finalShots to finalShots as string
set AppleScript's text item delimiters to ASCII character 10
set finalListoftapes to finalListoftapes as string
set finalListoftapes to finalShots & (ASCII character 10) & (ASCII character 10) & finalListoftapes & (ASCII character 10) & (ASCII character 10) & "This list of tapes has been saved on your desktop."
set path_list to these_items as string
set filepath to path_list & "_listoffiles.txt"
set outputFile to (open for access file filepath with write permission)
write finalListoftapes to outputFile
close access outputFile
display dialog finalListoftapes
end tell
end open

Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

Exception while Read very large file 300 MB

Hi
I want to read a file which has size of more than 300 MB.
While i m executing my code, i m getting
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Below is my code
1. FileChannel fc = new FileInputStream(fFile).getChannel();
2.CharBuffer chrBuff = Charset.forName("8859_1").newDecoder().decode(fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size()));
3.fc.close();Im getting the exception in the line 2.
Even though i increased the Heap space up to 900 Mb, i m getting this error.
(FYI)
I executed in command prompt like below,
java -Xms128m -Xmx1024m readLargeFile
Kindly give a solution for this. is there any other best way to read large text file ??
I m waiting for your reply..
Thanks in advance

Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.
The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.
Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.
Thanks in Advance...

What file type should i use for reading a large file of data?

I want to store data in a file and access them later. The data file could reach 500 to 1000 MBytes, as a binary file.
I may have memory problems.
I do not need the whole data in memory, only a few data each time, for my calculations.
If i use a binary file, i can not read only a few data. I have to load the whole file and then read the data i need.
Is this correct?
If i use another type of file, can i read only a few bytes without loading the whole file?
Maybe a TDMS file?
Solved!
Go to Solution.

I would probably use a TDMS file for this since it could also be read into excel if it were small enough - just more flexiblility. But you can also do it using binary types. You don't have to read the entire file when using binary files. See below.

I need help on reading a large file, thanks.

I try to read a txt file into an array, just like what I ususally do. But this time the file is too big (300k rows, 100 columns seperated by single space), I got the message "java.lang.OutOfMemoryError".
I need to do some calculation for every possible row pair. Can someone help me out of this? Thanks a lot.

Try changing your approach - don't read it into an array, process it one line at a time. Obviously any approach where you have the whole file in memory is going to exceed memory at some size of the file.

JEditorPane reading in large files

Hello,
I am currently using JEditorPane.read() to read a rather big ASCII text file. It works for the most part except:
1. The java runtime dies due to out of memory exception for any file that is over around 6 megabytes in size.
2. Any file whose size is around 5.5 - 6 megabytes tables around a minute to load. I would like to cut down on the loading time.
Does anyone have any suggestions? Is there an alternative besides using JEditorPane as a view into a large log file? I am using it as view only and not editing any data with it.
Thanks
Jim

my collegue has a similar problem.
have you fixed this yet ??
do you use fileInputStream or as suggested in various
replies i've seen use the BufferedFileInPutStream.
hope this helps,,,, let me know your status
thanks
Rob.

How to read a large file(size around 100 kbytes) into a byte array in JAVA

aa

But I cannot use array subscript as a long value.what do you mean?
are you rying the following:
long aSize = 100000;
byte[] array = new byte[aSize];
if this is the case, then don't do it, for your 100k file int is far more than enough.
but in case you jsut get error while:
byte b = array[100000];
then it's probably because you'r array is of size 100000, that means you have bytes from 0 to 99999 in there, and 100000 is one passed the end of the array.... jus in case make your array size to be 1234567 (far more than needed for your file) and see if you still get that exception, if not, then your array was too small. but if it fixes your prob, then don't leave it as it is, try to work out the right size for array and use that....
i wouldn't recommend using static array size, since that file might some day be bigger than you expected, and then you'd have to recompile your program.
HTH

UTL_FILE.get_line won't read large files ?

I am trying to read a large fixed length flat file. If I cut the file down to really small it will read it but it reads it as a single line. If I try to read a larger file > 32k I get a READ_ERROR. I am pretty sure it has to do with the end of line marker but I saw nothing about that in the UTL_FILE documentation. This is on UNIX, new line character after each record in the file. Standard unix flat file.
Any ideas on what to do?
Thanks in advance
Matt
[email protected]
my code:
BEGIN
BEGIN
std_file := UTL_FILE.FOPEN('&4','&1','r',32767);
EXCEPTION
WHEN UTL_FILE.INVALID_PATH THEN
RAISE_APPLICATION_ERROR(-20011,'Invalid Path for STD file, &4/&1');
WHEN OTHERS THEN
RAISE_APPLICATION_ERROR(-20014,'Other Error trying to open STD file, &4/&1');
END;
IF UTL_FILE.is_open(std_file) = FALSE THEN
RAISE_APPLICATION_ERROR(-20015,'Could not open STD file, &4/&1');
END IF;
-- READ STD FILE HEADER
BEGIN
UTL_FILE.get_line(std_file,hdr_text);
EXCEPTION
WHEN UTL_FILE.INVALID_FILEHANDLE THEN
RAISE_APPLICATION_ERROR(-20017,'STD read file handle not valid');
WHEN UTL_FILE.INVALID_OPERATION THEN
RAISE_APPLICATION_ERROR(-20018,'STD read invalid operation error');
WHEN UTL_FILE.READ_ERROR THEN
RAISE_APPLICATION_ERROR(-20019,'STD read error');
WHEN NO_DATA_FOUND THEN
RAISE_APPLICATION_ERROR(-20020,'STD read no data found');
WHEN VALUE_ERROR THEN
RAISE_APPLICATION_ERROR(-20021,'STD read value error');
END;
-- PROCESS TRANSACTIONS
LOOP
BEGIN
tran_text := NULL;
UTL_FILE.get_line(std_file,tran_text);
EXCEPTION
WHEN no_data_found THEN EXIT; -- EOF
WHEN value_error THEN
RAISE_APPLICATION_ERROR(-20010,'STD record too long.');
END;
std_rowcount := std_rowcount + 1;
END LOOP;
UTL_FILE.FCLOSE(std_file);
EXCEPTION
WHEN NO_DATA_FOUND THEN
RAISE_APPLICATION_ERROR(-20001,'No Data Found.');
WHEN UTL_FILE.INVALID_PATH THEN
RAISE_APPLICATION_ERROR(-20002,'Invalid Path ');
WHEN UTL_FILE.INVALID_MODE THEN
RAISE_APPLICATION_ERROR(-20003,'Invalid Mode ');
WHEN UTL_FILE.INVALID_OPERATION THEN
RAISE_APPLICATION_ERROR(-20004,'Invalid Operation ');
END;
null

We are still hung up on this. I tried implementing the code from STEVE'S XML book but still haven't resovled it.
The clob is being created via XSU see below. The new char[8192] appeasr to force the output file to 8K
with trailing characters on small clobs but adds a carraige return each 8K on larger ones.
As usual any input is appreciated from all. Doese anyone know of a good JAVA forum like this one?
Thanks
PROCEDURE BuildXml(v_return OUT INTEGER, v_message OUT VARCHAR2,string_in VARCHAR2,xml_CLOB OUT NOCOPY CLOB) IS
queryCtx DBMS_XMLquery.ctxType;
Buffer RAW(1024);
Amount BINARY_INTEGER := 1024;
Position INTEGER := 1;
sql_string VARCHAR2(2000) := string_in;
BEGIN
v_return := 1;
v_message := 'BuildXml completed succesfully.';
queryCtx := DBMS_XMLQuery.newContext(sql_string);
xml_CLOB := DBMS_XMLQuery.getXML(queryCtx);
DBMS_XMLQuery.closeContext(queryCtx);
EXCEPTION WHEN OTHERS THEN
v_return := 0;
v_message := 'BuildXml failed - '||SQLERRM;
END BuildXml;
create or replace and compile java source named sjs.write_CLOB as
import java.io.*;
import java.sql.*;
import java.math.*;
import oracle.sql.*;
import oracle.jdbc.driver.*;
public class write_CLOB extends Object
public static void pass_str_array(oracle.sql.CLOB p_in,java.lang.String f_in)
throws java.sql.SQLException, IOException
File target = new File(f_in);
FileWriter fw = new FileWriter(target);
BufferedWriter out = new BufferedWriter(fw);
Reader is = p_in.getCharacterStream();
char buffer[] = new char[8192];
int length;
while( (length=is.read(buffer)) != -1) {
out.write(buffer);
is.close();
fw.close();
/

A usefull way to read a long file

Hi,
What's the best and usefullest way to read a large file?
I do use a LineNumberReader but i think it reads a little slow...
To open the same file in an text editor reacts in less a millisecond while my LineNumberReader nees more time to read the entire file...
Any suggestions?
Olek

re,
how can i determine how a file contains "real" text or only sensles characters?
The file i readed seems to be a binary file in your meaning.
here is the method i use
// Threader.FileReadThreader is a instance of SwingWorker
public static String readPlainTextFile(File f, Threader.FileReadThreader fileReader) throws IOException {
        final StringBuilder strBuild = new StringBuilder();
        final DataInputStream dis = new DataInputStream(new BufferedInputStream(new FileInputStream(f)));
        final int readByte = BUFFER;
        final int fileLength = (int) f.length();
        float step = fileLength / readByte;
        byte[] buffer = new byte[fileLength];
        int off = 0;
        if (fileReader != null) {
            float percent = 0F;
            while (off < fileLength - readByte) {
                dis.readFully(buffer, off, readByte);
                String tempSequence = new String(Arrays.copyOfRange(buffer, off, off + readByte - 1));
                strBuild.append(tempSequence);
                off += readByte;
                percent += step;
                fileReader.doPublish(new Object[]{Math.round(percent), tempSequence});
            // read the remaining bytes
            int last = fileLength - off;
            dis.readFully(buffer, off, last - 1);
            String tempSequence = new String(Arrays.copyOfRange(buffer, off, off + last - 1));
            strBuild.append(tempSequence);
            percent += step;
            fileReader.doPublish(new Object[]{Math.round(percent), tempSequence});
            return strBuild.toString();
        } else {
            dis.readFully(buffer);
            return new String(buffer);
    }As you can see i modified your code.
Hm i got the error why the JTextArea do only display
one large line of characters...
i forgot to set line wrap ...
but now the app freezes reading the large binary file ...
with a smaller file all works ok.
Any ideas?
regards.
Olek

Dealing with large files, again

Ok, so I've looked into using BufferedReaders and can't get my head round them; or more specifically, I can't work out how to apply them to my code.
I have inserted a section of my code below, and want to change it so that I can read in large files (of over 5 million lines of text). I am reading the data into different arrays and then processing them. Obvioulsy, when reading in such large files, my arrays are filling up and failing.
Can anyone suggest how to read the file into a buffer, deal with a set amount of data, process it, empty the arrays, then read in the next lot?
Any ideas?
void readV2(){
     String line;
     int i=0,lineNo=0;
        try {
          //Create input stream
          FileReader fr = new FileReader(inputFile);
             BufferedReader buff = new BufferedReader(fr);
            while((line = buff.readLine()) != null) {
          if(line.substring(0,2).equals("V2")){
                 lineNo = lineNo+1;
          IL[i] = Integer.parseInt(line.substring(8,15).trim());
                //Other processing here
                 NoOfPairs = NoOfPairs+1;
                 }//end if
                 else{
                  break;
        }//end while
        buff.close();
        fr.close();
        }//end try
        catch (IOException e) {
        log.append("IOException error in readESSOV2XY" + e + newline);
        proceed=false;
        }//end catch IOException
        catch (ArrayIndexOutOfBoundsException e) {
               arrayIndexOutOfBoundsError(lineNo);
     }//end catch ArrayIndexOutOfBoundsException
     catch (StringIndexOutOfBoundsException e) {
          stringIndexOutOfBoundsError(e.getMessage(),lineNo);
}//end V2Many thanks for any help!
Tim

Yeah, ok, so that seems simple enough.
But once I have read part of the file into my program,
I need to call another method to deal with the data I
have read in and write it out to an output file.
How do I get my file reader to "remember" where I am
up to in the file I'm reading?
An obvious way, but possibly not too good technically,
would be to set a counter and when I go back to the
fiel reader, skip that number of lines in the inpuit
file.
This just doesn't seem too efficient, which is
critical when it comes to dealing with such large
files (i.e. several million lines long)I think you might need to change the way you are thinking about streams. The objective of a stream is to read and process data at the same time.
I would recommend that you re-think your algorithm : instead of reading the whole file, then doing your processing - think about how you could read a line and process a line, then read the next line, etc...
By working on just the pieces of data that you have just read, you can process huge files with almost no memory requirements.
As a rule of thumb, if you ever find yourself creating huge arrays to hold data from a file, chances are pretty good that there is a better way. Sometimes you need to buffer things, but very rarely do you need to buffer such huge pieces.
- K

Adobe AIR 4 Not reading a XML file larger than 10MB

I have an application where the XML file could be over 10MB in size. I am using XMLHttpRequest to read the file in. However, on Air 4, it results in null when the file is greater than 10MB in size. This was working until I updated to AIR 4. I can go back to a previous version of AIR and the loading of large file works.
xml = new XMLHttpRequest( );
xml.onreadystatechange = function( ){
if( xml.readyState == 4 && xml.status == 200){
b[language] = xml.responseXML.documentElement; //Results in NULL when file size is greater than 10MB
var tempTxt = "Loaded Database";
xml.open( "GET", file.url, false);
xml.send( null );
Why AIR limited the file size and any work around for this? The Request processes fine and readyState will become 4. However, xml.responseXML.documentElement will respond with a NULL.
Thank you
Binu
www.verseview.info

It's difficult to tell what it is, it looks like a binary
pipe symbol but I can copy it from TextPad for example. Some of the
characters following it cannot be copied from TextPad which I
assume is because it's null. I can read the whole file in C#/.Net
and assign it to a string variable without any problems but perhaps
Air is somewhat limited to binary content.

Cannot transfer large files between drives - data can't be read or written

Hi there!
I am trying to transfer a couple of large files to a hard drive from my internal ssd. Both are set to Mac OS Extended (Journaled). Small files transfer fine, but large ones (eg. 20GB) get a varying distance through and then the following error message comes up:
The Finder can't complete the operation because some data in “FILENAME” can’t be read or written. (Error code -36)
I have tried to ensure read&write permissions are allowed, but this hasn't helped. This has occurred with multiple files.
Any help would be very much appreciated.
Max

Hi
I exported a movie
How ?
One must/can only move iMovie Events and/or Projects - WITHIN IMOVIE !
Do not alter or move anything in
• iMovie Project folder - or -
• iMovie Events folder
IMovie get's confused - and damage might well be un-repairable !
How was the target hard disk formatted ?
Mac OS Extended - only option that works !
How much space was there on the target hard disk ?
1 hour SD-video needs about 13Gb
HD about 5 times more
I would
• Connect second laptop (Mac) via FireWire
• Start this second one - Keeping T-key down (target mode)
• Now in iMovie Project resp Events window it will show up as an external hard disk
Drag-and-drop Events and/or projects over.
Yours Bengt W

Reading A Large File At Launch

Similar Messages

Maybe you are looking for