Reading large graphics files as manageable blocks?

I am looking either for source code or a library that would allow me to easily read a graphics file in manageable blocks. For example given an image that is 20000x200000 pixels, I would like to be able to read only a block of 400x400, from a given offset x/y in the image.
I need this since I am writing a program that needs to handle very large images (I can't assume user has more memory than the files), but does not need to display the whole image in one go, and but would let the user navigate within the image. Formats I am looking to have support with are JPEG, PNG and TIFF.

I agree with Gerd.  If you are working with binaries, why not use U8 instead of doubles.
If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000.  So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration).  This is just speculation since I don't see the portion of your code that populates the array.
Question:  Why do you need an array?  What do you do with the data after you read it?  I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

Similar Messages

  • A pdf file failed to convert to word, presumably because of size.  how do i split a large pdf file into manageable secrtions?

    I'm running Abode Reader XI version 11.0.7.  Repeated attempts to convert a large (439 page) file, a dissertation, failed.  How do I split a large pdf file like this into manageable sections for conversion?

    Hi Mike,
    Your 11MB file is well within the file-size limits for ExportPDF, but depending on the number of pages, complexity of the file (and yours doesn't sound complex), and your connection speed, it is possible that the service is simply timing out before it can finish processing. These steps can help:
    If the file already contains editable text (that is, it isn't a scanned document), try disabling OCR as outlined in this this document: How to disable Optical Character Recognition (OCR) when converting PDF to Word or Excel.
    Clear the browser cache and try again.
    Try a different browser.
    Let's start there. If you still can't export the file to Word, let me know and we'll take it from there.
    Best,
    Sara

  • Reading Large data files from client machine ( urgent)

    Hi,
    I want to know the best way for reading large data at about 75MB file from client machine and insert to the database.
    Can anybody provide sample code for it.
    Loading the file should be done at client machine and inserting into the database should be done at server side.
    How should i load the file?
    How should i transfer this file or data to server ?
    How should i insert into the database ?
    Thanks in advance.
    regards
    Kalyan

    Like I said before you should be using your application server to serve files >from the server off the filesystem. The database should not store files this big >and should instead just have a reference to this file. I think u have not understood the problem corectly.
    I will make it clear.
    The requirement is as follows.
    This is a j2ee based application.
    Application server is oracle application server.
    Database is oracle9i
    it is thick client (swing based application)
    User enters datasource like c:\turkey.data
    This turkey.data file contains data
    1@1@20050131@1@4306286000113@D00@32000002005511069941@@P@10@0@1@0@0@0@DK@70059420@4330654016574@1@51881100@51881100@@99@D@40235@0@0@1@430441800000000@@11@D@42389@20050201@28483@15@@@[email protected]@@20050208@20050307@0@@@@@@@@@0@@0@0@0@430443400000800@0@0@@0@@@29@0@@@EUR
    like wise we may have more than 3 lacs rows in it.
    We need to read this file and transfer this to the application server. Which are EJBS.
    There we read this file each row in file is one row in the database for a table.
    Like wise we need to insert 3 lacs records in the database.
    We can use Jdbc to insert the data which is not a problem.
    Only problem is how to transfer this data to server.
    I can do it in one way. This is only a example
    I can read all the data in StringBuffer and pass to server.
    There again i get the data from StringBuffer and insert into database using jdbc.
    This way if u do it. It is performance issue and takes long time to insert into the database.It even may give MemoryOutofBond exception.
    just iam looking for the better way of doing this which may get good performace issue.
    Hope u have understood the problem.

  • How to read LARGE .TXT FILES from server

    i m unable to read large .txt (>100kb) files from server.
    also is there any way to store the content in mobile phone.
    PLEASE HELP as iam in dire need and only few days are left.
    thanks

    Hi
    > i m unable to read large .txt (>100kb) files from server.
    What do you mean by that? Are you trying to display that amount of text and you get an error. Are you trying to read from the server and you get there an error?
    > also is there any way to store the content in mobile phone.
    Using RMS. But be aware of the size limit specific to every phone.
    Mihai

  • Read large binary file

    How would you read a large binary file into memory please? I had a thought about creating a Byte Array on the fly, however you cannot createa byte array with a long so what happens when you reach the maximum value an integer can store?

    a) You can map the file, instead of reading it physically.
    b) Let's suppose that you are running Sun JVM in Windows 2000/XP/2003 and you have 4-GB of RAM in your machine (memory is so cheap nowadays...)
    - Windows can not use the full 4GB (it reserves 2GB for itself, and if you buy a special server version, it reserves only 1GB for itself.)
    - Java can't use more than 1.6 GB in Windows due to some arcane reasons. (Someone posted the Bug Database explanation in this forum.)
    So you have an upper limit of 1.6 GB.
    1.6GB is a value smaller than the size of an int, so you could have an array as big (using the class java.nio.ByteBuffer and the like). Try and see.

  • Reading large XML file using a file event generator and a JPD process

    I am using a FileEventGenerator and a JPD Subscription process to read a large XML file. The large XML file basically contains repeated XML elements. My understanding is that the file subscription method reads the whole file in memory which causes lots of problem for huge file size like 1MB. Is there a way to read the file size-wise or is there a way to read chunks of data from a large size file..or any other alternative. I would like to process the file in a loop iteration by iteration.

    Hitejain,
    Here are a couple of pointers you could try. One is that the file event generator has a pass by reference (filename) functionality which you could use so that you could do the following inside of your process.
    1) Read file name from the reference
    2) Move the file to a processed directory (so it doesn't get picked up again. Note: I don't know how the embedded archive methods of the file event generator plays with pass by reference.
    3) Open a stream to the file.
    4) Use a SAX or SAX - DOM combined approach to parse your XML while managing the memory usage inside of your process
    There is another possibility which might fit your needs and it is related to the RawData object that BEA provides. If I understand it correctly provides wrapping functionality around a stream object, but depending on your parsing methods might just postpone the problem.
    Hope this helps
    Chris Falling
    Stormforge Software

  • Reading Large XML Files

    Hi All,
    I am trying to read an XML file which is around 6.4 GB which has around 1000 million records in it.
    I need to travel through all the elements and take out the values which matches my criteria.
    For Example the XML Looks like :
    <ss>
    <s>
      <m>00:11:22:33:44:55</m>
      <o>1244548353223</o>
      <f>1244548354223</f>
    </s>
    <s>
      <m>00:11:22:33:44:56</m>
      <o>1244548353223</o>
      <f>1244548354223</f>
    </s>
    </ss>I need to get the <s> node which has the <m> value is 00:11:22:33:44:55. It may have around 1024 occurences per file.
    I tried XPath, XQuery apis for java and nothing helped me in reading such a huge file.
    The actual task is to read 2 to 30 huge files concurrently.
    Can anyone help me in this?
    Thanks in advance.
    Kousik Rajendran

    I tried XOM. It worked fine for 102.4 MILLION data stored in a xml file which has size about 5.5GB.
    Checkout [XOM Home Site|http://www.xom.nu/]
    Or dowload Zip File with all necessary Zip and Sample file : [XOM Complete Zip|http://www.cafeconleche.org/XOM/xom-1.2.1.zip]
    Try out the Sample which uses Streaming Method.

  • Signed Applet - Problem reading large local files

    Hello,
    I am using a signed applet (self-signed) to read files from a clients machine (local file system). Following is the code for buffered reading (the meat):
    String thisLine, ret = "";
    try{
         BufferedReader myInput = new BufferedReader(new FileReader(fn));
         while ((thisLine = myInput.readLine()) != null) {
         ret += thisLine + "\n";
         myInput.close();
    catch (Exception e) {
         ret = "Cannot load, exception!";
    When I read a file <= 512KB, it runs fine/as expected. Now, when I read a file, say for example 1024KB in size, I expect the application to take twice as much time (than 512KB file) and read the file, but instead the applet/browser hangs. As a matter of fact I am not able to read files > 512KB.
    So, is there a limit on file size (from local file system) when reading from an applet, or is it to do with the way I am reading it?
    Any help, suggestions or comments would be highly appreciated.
    Thanks in advance.

    Here is how I read a file from my applet:
         private String getData(String u) {
              URL url = null;
              InputStream in = null;
              String ret = null;
              try {
                   // the following line wil read a file @ the same location as your class file
                   // this way your applet does not need any special privileges
                   // you could change the url to a file if so needed
                   url = new URL(this.getCodeBase(), u);
                   URLConnection conn = url.openConnection();
                   // next line is optional and just tells the http server
                   // a utf-8 response is accepted
                   // some servers will return a bad request
                   conn.setRequestProperty("Accept-Charset","utf-8");
                   // first try to find out if the server has
                   // provided us with the charset of the response
                   // since this doesn't work with the msjvp (version 1.1.2)
                   // it's commented out
                   // instead provide a charEncoding yourselve
                   String charEncoding = "UTF8";
    //               int i = conn.getHeaderFields().size()-1;
    //               while(i>-1){
    //                    System.out.print(conn.getHeaderFieldKey(i));
    //                    System.out.print("=");
    //                    System.out.println(conn.getHeaderField(i));
    //                    if(conn.getHeaderField(i).toLowerCase().indexOf("charset")!=-1){
    //                         String charsetLine = conn.getHeaderField(i).toUpperCase();
    //                         if(charsetLine.indexOf("SHIFT_JIS")!=-1){
    //                              charEncoding = "Shift_JIS";
    //                         if(charsetLine.indexOf("EUC-JP")!=-1){
    //                              charEncoding = "EUC-JP";
    //                    i--;
                   // here is the value for char encoding (as a parameter in the <object html tag)
                   if(this.getParameter("dataCharset")!=null){
                        charEncoding=this.getParameter("dataCharset");
                   // wither you have a file or an url you read from an inputstream
                   // here is how it's done
                   // read the whole file and than turn it into a string
                   in = conn.getInputStream();
                   ByteArrayOutputStream bos = new ByteArrayOutputStream();
                   int len;
                   byte[] buf = new byte[1024];
                   while ((len = in.read(buf)) > 0) {
                        bos.write(buf, 0, len);
                   in.close();
                   // you have the file as a byte array (bos.toByteArray()
                   // here is how you turn it into a string
                   ret = new String(bos.toByteArray(),charEncoding);
              } catch (Exception e) {
                   e.printStackTrace(System.out);
              return ret;
         }

  • Reading Large Matlab files into LabVIEW

    Hi All,
    I am using the 'Open Data Storage [MATLAB] block to read data from a mat file into MATLAB. The data is very large (in the order of millions). When I try to run my vi which is relatively simple to read in and manipulate the data, my LabVIEW said it does not have enough memory. Is there any way to around this problem without physically increasing computer memory.
    Thanks in advance for any help and assistance.
    Chris

    Hi Chris,
    Did the 'not enough memory' happen on 'Open Data Storage' or 'Read Data' ?
    If it happened on the 'Read Data', you might want to try specifying 'index/count' inputs of the 'Read Data' to read channel values chunk by chunk.
    Best Regards,
    Mavis

  • Reading large binary files into an array for parsing

    I have a large binary log file, consisting of binary data separted by header flags scattered nonuniformly thorughout the data.  The file size is about 50M Byte.  When I read the file into an array, I get the Labview Memory full error.  The design of this is to read the file in and then parse it fro the flags to determine where to separate the data blocks in the byte stream. 
    There are a few examples that I have read on this site but none seem to give a straight answer for such a simple matter.   Does anyone have an example of how I should approach this?

    I agree with Gerd.  If you are working with binaries, why not use U8 instead of doubles.
    If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000.  So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration).  This is just speculation since I don't see the portion of your code that populates the array.
    Question:  Why do you need an array?  What do you do with the data after you read it?  I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

  • Texhnique to read large files (20-200Mb) using IO

    Hi,
    Can anyone send me code to read large XML files from Network or local machine say 20-200Mb, when i use normal IO it takes huge time.
    I need some optimum or buffered approach.
    Thanks and Regards
    Amit

    I am sorry, i don't know why you are so rude i am not giving u any homework assigned to me, i am just a new learner and i also don't like any spoonfeeding but i am a c++ programmer, and just learning java
    I am getting error
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    when i am using xerces 2.9.0 JDK1.5,
    The XML file size is approx 1G
    I am using
    try
    File file = new File(m_szFileName);
    DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
    DocumentBuilder db = dbf.newDocumentBuilder();
    Document doc = db.parse(file);
    doc.getDocumentElement().normalize();
    m_rootNode = doc.getDocumentElement();
    catch (Exception e)
    e.printStackTrace();
    when i get document.parse then it gives error
    Document doc = db.parse(file);
    I think this loads whole file into heap and that heap is not sufficient
    This code is running if i put upto 5Mb file.
    I want to know whether it uses internally some buffering mechanism or that i need to do externally.
    if yes for that i am asking for sample codes.
    Also i got some solutions like
    java -Xms<initial heap size> -Xmx<maximum heap size>
    Defaults are:
    java -Xms32m -Xmx128m
    but i think this is not a proper solution to increase heap size, i don't know but according to me it's wrong approach to increase heap according to requirement
    specially when i don't know max size of xml, it can be upto 4G, or like i can also need to read eventviewer XML saved file whole max limit is 4G.
    Also i come to know bug in
    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6536111
    jre1.6,
    but i think i am using jre1.5
    b'coz my jdk version is 1.5
    Can you guide me
    Thanks in advance
    amit3281
    Message was edited by:
    amit3281
    Message was edited by:
    amit3281

  • Lion server file sharing issue with windows API read/write ini file (GetPrivateProfileString)

    Hello,
    I try to config lion server as file server for a windows application we use at work. All other computers are windows 7 or XP, lion server is the only mac. I choose lion server because it's size, quality and personal love of apple products.
    10.7.2 lion server's samba file sharing works almost perfectly with all my windows machines, I can copy, delete, modify any text files or office files without any issue, but the most important windows application for my business doesn't work with samba file sharing. After some digging, I found it is because windows program can't read or write INI file stored on lion share. Windows API GetPrivateProfileString always returns empty if the INI file is store on lion share.
    You can download a small application for read/write windows INI file from codeproject.com to test this problem:
    http://www.codeproject.com/KB/files/ini.aspx
    I can open/edit the in file using any text editor without any problem. The only problem is with those windows APIs. ACL is turned on for my lion share and assigned "delete" rights to samba users.
    I install samba3 on the same server; it works perfectly with windows API. My windows program also works. Looks like there is something wrong with lion server's sambax.
    I'd prefer to use built-in samba even I have samba3 working. Built-in samba is very immature right now, but considered how young it is, I will give apple some time to make it mature.
    Does anyone have same issue or knows how to fix it?
    Thanks,
    Michael.

    All the memory is fine. The server rarely if ever goes down when there are only around 10-12 users connected. When there are 20+ users connected and working heavily it goes down often. When I say working heavily, I mean they are transferring huge files to the SAN (100GB+), sometimes 5 at a time per user, and there are a bunch of others who are reading large video files at a minimum of 220MB/sec from the SAN.
    Though this worked on Snow Leopard without any issues, Lion just doesn't seem to be able to handle it. The odd thing is, on Snow Leopard there was only a single 1GB ethernet connection to a NAS system, whereas with Lion we have a much more powerful machine with a 6-port 10GB ethernet card and a 4 lane 8GB fiber card to a true SAN. You would think that the newer scenario with Lion would handle far more users with ease.
    So far, very disappointing with regards to Lion's file serving performance.

  • Color printing help with large graphics Reader 11.0.3

    Good morning!  We recently upgraded to 11.0.3.  Windows 7 64-bit.  We've had issues printing large color graphics from Reader 11.
    For example, the PDF will have an intricate purple graphic with matching purple text blocks underneath.  Print preview shows correctly, but when printed the graphic prints brown instead of purple.  The purple text blocks print correctly.  This seems to happen sporadically, but seems to be with complex graphics.
    Documents from all other apps print OK and if we open this .pdf from Acrobat X Standard and export it to a .png file the colors print correctly. 
    The graphic files come from a myriad of sources and customers and we don't have control over the way the documents are originally saved or what apps are used to create the artwork. 
    Any idea why the graphics colors aren't printing and if there are settings we can adjust in Reader 11 so they will print correctly?
    We've tried:
    - Printing to several printers with different drivers and all yield the same results.  (Toshiba eStudio 3530c with PCL6 and PS drivers, Epson Artisan 835, HP OfficeJet 8600, Samsung CLX-6200x)
    - Adjusting the Overprint Preview settings under File -> Preferences and setting "Simulate Overprinting" under the Advanced button in the print preview window.
    - Selecting "Print As Image" under the Advanced button in the print preview window.
    - Adjusting the Document Color Options under File -> Preferences -> Accessibility.
    Thanks in advance for any help!

    Fastest thing you can try: use Windows Explorer to navigate to C:\Program Files (x86)\Adobe\Reader 11.0\Reader, then double-click on Eula.exe and accept the license agreement.
    If that doesn't work, you can try using this tool to first remove all traces of Reader from your computer:
    http://labs.adobe.com/downloads/acrobatcleaner.html
    Then, you can download the full offline Reader installer from
    http://get.adobe.com/reader/enterprise/
    After downloading, restart your computer and run the installer before doing anything else.

  • Reading, filtering and playing large WAV files

    Hi!
    I want to build an application that reads LARGE wavefiles, filters them and plays them via the soundcard.
    I tried to build something with the standard blocks available in Labview 7.0, but I keep running out of memory. The files I want to filter and play are simply too large The filesize is typ. 650Mb = 1 hour of stereo CD quality audio. The files I am using just won't play even on a WIN XP machine with 1Gb of memory.
    Next step was to build a function that loads the WAV files in blocks of ~16kB. This works fine, but now the filtering does not work anymore; after each block, the 'FIR Filter' block seems to reset its states. The result is a very bad repeating artifact in the sound. The repetition-rate is related to the buffer size.
    Does anyone have experience with this type of application? Help and examples are -very- welcome (as I am new to Labview... .
    Many thanks,
    Dr.dB

    I have created a VI that will do the continuous FIR filtering you need. I assume you want to filter a 'stereo' signal so the VI takes an array of waveforms as input. In this version the FIR filter coefficients (kernel) is common to all channels. If this is not what you need you can correct the VI accordingly.
    The control 'algorithm' specifies whether the FIR convolution is internally performed in the time or frequency domain. Changing the 'algorithm' does not affect your result but affects your performances. Select time domain (direct) if your kernel size (number of FIR coefficients) is, say less than 100, otherwise select 'frequency domain'.
    Try the attached example and let me know if you have any problems.
    Attachments:
    continuous FIR filtering on multiple channels.llb ‏79 KB

  • Adobe AIR 4 Not reading a XML file larger than 10MB

    I have an application where the XML file could be over 10MB in size. I am using XMLHttpRequest to read the file in. However, on Air 4, it results in null when the file is greater than 10MB in size. This was working until I updated to AIR 4. I can go back to a previous version of AIR and the loading of large file works.
    xml = new XMLHttpRequest( );
    xml.onreadystatechange = function( ){
        if( xml.readyState == 4 && xml.status == 200){
      b[language] = xml.responseXML.documentElement; //Results in NULL when file size is greater than 10MB
            var tempTxt = "Loaded Database";
    xml.open( "GET", file.url, false);
    xml.send( null );
    Why AIR limited the file size and any work around for this? The Request processes fine and readyState will become 4. However, xml.responseXML.documentElement will respond with a NULL.
    Thank you
    Binu
    www.verseview.info

    It's difficult to tell what it is, it looks like a binary
    pipe symbol but I can copy it from TextPad for example. Some of the
    characters following it cannot be copied from TextPad which I
    assume is because it's null. I can read the whole file in C#/.Net
    and assign it to a string variable without any problems but perhaps
    Air is somewhat limited to binary content.

Maybe you are looking for