Reading large files -- use FileChannel or BufferedReader?

Question --
I need to read files and get their content. The issue is that I have no idea how big the files will be. My best guess is that most are less than 5kb but some with be huge.
I have it set up using a BufferedReader, which is working fine. It's not the fastest thing (using readLine() and StringBuffer.append()), but so far it's usable. However, I'm worried that if I need to deal with large files, such as a PDF or other binary, BufferedReader won't be so efficient if I do it line by line. (And will I run into issues trying to put a binary file into a String?)
I found a post that recommended FileChannel and ByteBuffer, but I'm running into a java.lang.UnsupportedOperationException when trying to get the byte[] from ByteBuffer.
File f = new File(binFileName);
FileInputStream fis = new FileInputStream(f);
FileChannel fc = fis.getChannel();
// Get the file's size and then map it into memory
int sz = (int)fc.size();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, sz);
fc.close();
String contents = new String(bb.array()); //code blows up
Thanks in advance.

If all you are doing is reading data I don't think you're going to get much faster than InfoFetcher
you are welcome to use and modify this class, but please don't change the package or take credit for it as your own work
InfoFetcher.java
==============
package tjacobs.io;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Iterator;
* InfoFetcher is a generic way to read data from an input stream (file, socket, etc)
* InfoFetcher can be set up with a thread so that it reads from an input stream
* and report to registered listeners as it gets
* more information. This vastly simplifies the process of always re-writing
* the same code for reading from an input stream.
* <p>
* I use this all over
     public class InfoFetcher implements Runnable {
          public byte[] buf;
          public InputStream in;
          public int waitTime;
          private ArrayList mListeners;
          public int got = 0;
          protected boolean mClearBufferFlag = false;
          public InfoFetcher(InputStream in, byte[] buf, int waitTime) {
               this.buf = buf;
               this.in = in;
               this.waitTime = waitTime;
          public void addInputStreamListener(InputStreamListener fll) {
               if (mListeners == null) {
                    mListeners = new ArrayList(2);
               if (!mListeners.contains(fll)) {
                    mListeners.add(fll);
          public void removeInputStreamListener(InputStreamListener fll) {
               if (mListeners == null) {
                    return;
               mListeners.remove(fll);
          public byte[] readCompletely() {
               run();
               return buf;
          public int got() {
               return got;
          public void run() {
               if (waitTime > 0) {
                    TimeOut to = new TimeOut(waitTime);
                    Thread t = new Thread(to);
                    t.start();
               int b;
               try {
                    while ((b = in.read()) != -1) {
                         if (got + 1 > buf.length) {
                              buf = IOUtils.expandBuf(buf);
                         int start = got;
                         buf[got++] = (byte) b;
                         int available = in.available();
                         //System.out.println("got = " + got + " available = " + available + " buf.length = " + buf.length);
                         if (got + available > buf.length) {
                              buf = IOUtils.expandBuf(buf, Math.max(got + available, buf.length * 2));
                         got += in.read(buf, got, available);
                         signalListeners(false, start);
                         if (mClearBufferFlag) {
                              mClearBufferFlag = false;
                              got = 0;
               } catch (IOException iox) {
                    throw new PartialReadException(got, buf.length);
               } finally {
                    buf = IOUtils.trimBuf(buf, got);
                    signalListeners(true);
          private void setClearBufferFlag(boolean status) {
               mClearBufferFlag = status;
          public void clearBuffer() {
               setClearBufferFlag(true);
          private void signalListeners(boolean over) {
               signalListeners (over, 0);
          private void signalListeners(boolean over, int start) {
               if (mListeners != null) {
                    Iterator i = mListeners.iterator();
                    InputStreamEvent ev = new InputStreamEvent(got, buf, start);
                    //System.out.println("got: " + got + " buf = " + new String(buf, 0, 20));
                    while (i.hasNext()) {
                         InputStreamListener fll = (InputStreamListener) i.next();
                         if (over) {
                              fll.gotAll(ev);
                         } else {
                              fll.gotMore(ev);
     }

Similar Messages

  • Read Large Files Using BPEL File Adapter

    Hi,
    I have a scenario, where files of large size in text format are to be read and send to 3rd party. MTOM Policy has to be attached. Files of size 3MB or less are polled and size greater than 3 MB are not Retrieved. How can I resolve the issue. Do I have to break the file and send the data? If so how?
    Thanks
    Ranga

    You have to use streaming feature of the JCA file adapter for handling huge file.
    You could go through the following link
      http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_file.htm#CIAHHEBF

  • File Adapter Not Reading Large Files

    Dear Experts,
    Enviornment :-
    OS:-Linux
    Jdeveloper:- 11.1.1.6
    SOA:-11.1.1.6
    Weblogic:-10.3.6
    JDK:-SUN
    Allocated RAM:-16GB
    Currently we are in UAT Phase and now we are facingan  issue in reading large file.Below is the Design details of the service
    FileAdapter(Read)-->Bpel(Business login,Using FlowN)-->FileAdapter(Write CSV),JMS Adapter(AQ JMS topics)
    In this case at the time of read itself we are facing issue. File adapter reading xml file but in receive activity receives input data as
    xmlDocKey:1C135990067411E3BFA6B5087B629F9DI
    I really couldn;t understand about the error. Even i tried reading as Opaque format and still end up with same error.
    In order to make sure i have create mediator and tried reading the file, in case i was able to read file upto 15mb with out any error.Also i tried as  "read as Attachment" in Bpel component and able to read the attachment upto 7 mb file, but this is hitting the performance side.
    I request some one please let me know why the file adapter is giving XmlDOCKey rather then XMLContent to the inputvariable
    Regards,
    Tarak

    Can you check your BPEL Properties in EM?
    Go to Soa-infra > right Click > SOA Administration > BPEL properties
    increase the Dispatcher Engine Thread = 10, Dispatcher Invoke Threads = 60 and Dispatcher Engine Threads to 90
    Click on "More BPEL Configuration Properties"
    Increase the  DispatcherMaxRequestDepth from 600 to 1000.
    Bounce the server and see if works..
    Bounce the server and try again.
    If this fails, try get a threshold by increasing the file size until it fails again. 

  • Reading large file with JCA Adapter in OSB

    Hello,
    We are searching for a solution how to read large file (>50M) from network drive and deliver it to queue via OSB 11gR4 (10.3.4). The problem is when reading the file with JCA File Adapter. It seems that it cannot handle as large files as we have. The documentation provides a way to bypass file size limitation by using Chunk Read but it seems to require BPEL Process execution which is not possible in our environment. Does anyone know if there are ways to implement this without having BPEL Process?
    Our usecase:
    read file from network drive -> transfer with OSB -> deliver MQ
    Other options than JCA File Adapter can be considered, if anyone can advice...

    If it's a plain routing use case and no message processing is required then you may simply use OSB's FILE transport instead of JCA adapter. Create a messaging type proxy service and select request message type as "binary". Also enable the content streaming (Disk buffer, compression).
    From OSB Dev guide -
    Oracle JCA Adapter for FTP and Files – Attachments (large payload support), pre- and post-processing of files, using a re-entrant valve for processing ZIP files, content streaming, and file chunked read are not supported with Oracle Service Bus.
    http://download.oracle.com/docs/cd/E17904_01/doc.1111/e15866/jca.htm#BABBICIA
    You may also refer -
    Reading huge flat file in OSB 11gR1
    Regards,
    Anuj

  • How to read pdf files using java.io package classes

    Dear All,
    I have a certain requirement that i should read and write PDF files at runtime. With normal java file IO reading is not working. Can any one suggest me how to proceed probably with sample code block
    Thanks in advance.

    hi I also have the pbm. to read pdf file using JAVA
    can any body help meWhy is it so difficult to read the thread you posted in? They say: java.io is pointless, use iText. So why don't you?
    or also I want to read a binary encoded data into
    ascii,
    can anybody give me a hint how to do it.Depends on what you mean with "binary encoding". ASCII's binary encoding, too, basically.

  • Reading XML file using BAPI and then uploading that xml file data into SAP

    I am getting a xml file from Java server. I need to take
    data from this file using BAPI and need to upload into SAP using SAP.
    Please tell me how to read XML files using BAPI's.

    <b>SDIXML_DATA_TO_DOM</b> Convert SAP data (elementary/structured/table types) into DOM (XML
    <b>SDIXML_DOM_TO_XML</b>  Convert DOM (XML) into string of bytes that can be downloaded to PC or application server
    <b>SDIXML_DOM_TO_SCREEN</b> Display DOM (XML)
    <b>SDIXML_DOM_TO_DATA</b>
    data: it_table like t001 occurs 0.
    data: l_dom      TYPE REF TO IF_IXML_ELEMENT,
          m_document TYPE REF TO IF_IXML_DOCUMENT,
          g_ixml     TYPE REF TO IF_IXML,
          w_string   TYPE XSTRING,
          w_size     TYPE I,
          w_result   TYPE I,
          w_line     TYPE STRING,
          it_xml     TYPE DCXMLLINES,
          s_xml      like line of it_xml,
          w_rc       like sy-subrc.
    start-of-selection.
      select * from t001 into table it_table.
    end-of-selection.
    initialize iXML-Framework          ****
      write: / 'initialiazing iXML:'.
      class cl_ixml definition load.
      g_ixml = cl_ixml=>create( ).
      check not g_ixml is initial.
      write: 'ok'.
    create DOM from SAP data           ****
      write: / 'creating iXML doc:'.
      m_document = g_ixml->create_document( ).
      check not m_document is initial.
      write: 'ok'.
      write: / 'converting DATA TO DOM 1:'.
      CALL FUNCTION 'SDIXML_DATA_TO_DOM'
        EXPORTING
          NAME               = 'IT_TABLE'
          DATAOBJECT         = it_table[]
        IMPORTING
          DATA_AS_DOM        = l_dom
        CHANGING
          DOCUMENT           = m_document
        EXCEPTIONS
          ILLEGAL_NAME       = 1
          OTHERS             = 2.
      if sy-subrc = 0.  write  'ok'.
      else.             write: 'Err =', sy-subrc.
      endif.
      check not l_dom is initial.
      write: / 'appending DOM to iXML doc:'.
      w_rc = m_document->append_child( new_child = l_dom ).
      if w_rc is initial.  write  'ok'.
      else.                write: 'Err =', w_rc.
      endif.
    visualize iXML (DOM)               ****
      write: / 'displaying DOM:'.
      CALL FUNCTION 'SDIXML_DOM_TO_SCREEN'
        EXPORTING
          DOCUMENT          = m_document
        EXCEPTIONS
          NO_DOCUMENT       = 1
          OTHERS            = 2.
      if sy-subrc = 0.  write  'ok'.
      else.             write: 'Err =', sy-subrc.
      endif.
    convert DOM to XML doc (table)     ****
      write: / 'converting DOM TO XML:'.
      CALL FUNCTION 'SDIXML_DOM_TO_XML'
        EXPORTING
          DOCUMENT            = m_document
        PRETTY_PRINT        = ' '
        IMPORTING
          XML_AS_STRING       = w_string
          SIZE                = w_size
        TABLES
          XML_AS_TABLE        = it_xml
        EXCEPTIONS
          NO_DOCUMENT         = 1
          OTHERS              = 2.
      if sy-subrc = 0.   write  'ok'.
      else.              write: 'Err =', sy-subrc.
      endif.
      write: / 'XML as string of size:', w_size, / w_string.
      describe table it_xml lines w_result.
      write: / 'XML as table of', w_result, 'lines:'..
      loop at it_xml into s_xml.
        write s_xml.
      endloop.
      write: / 'end of processing'.
    end of code
    Hope this will be useful.
    regards
    vinod

  • Reading XML file using BAPI  I must use adapters .

    Reading XML file using BAPI and then uploading that xml file data into SAP using BDC.
    I cant take file on to my Application server I am getting the file dynamically from other file server and I need to use BAPis to read data from XML file.please tell me what should be my Import,Export and Tables parameterrs should be.
    Thanks

    Hi,
    Import, export and table parameters for BAPI is required, without that BAPI will not able to collect the data from XML. What you need to do is write a Zprogram and collect the data, store that data in internal table and call the BAPI by passing required parameters.
    Different Scanarios:
    1) Before calling a BAPI write some other program which collects the data from XML and create a UNIX file. Try to get the data from UNIX FILE
    2) If you see the XML tags, data will be in side that tags, try to get the data from XML tags and store that data in one SAP table. You can use the BAPI by fetching the data from table
    3)Create a table and store the data in the table. Use the table in SAP to extract the data.
    BAPI won't work without any parameters, you have to pass some parameters then the BAPI will return some values.
    Hope i am clear.
    Thanks&Regards,
    -Suresh Revuru

  • Processing Large Files using Chunk Mode with ICO

    Hi All,
    I am trying to process Large files using ICO. I am on PI 7.3 and I am using new feature of PI 7.3, to split the input file into chunks.
    And I know that we can not use mapping while using Chunk Mode.
    While trying I noticed below points:
    1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
    Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
    2) I used Dummy Interface in my scenario and it worked Fine.
    So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
    Thanks in Advance,
    - Pooja.

    Hello,
    While trying I noticed below points:
    1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
    Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
    2) I used Dummy Interface in my scenario and it worked Fine.
    So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
    According to this blog:
    File/FTP Adapter - Large File Transfer (Chunk Mode)
    The following limitations apply to the chunk mode in File Adapter
    As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following limitations would apply
    Only for File Sender to File Receiver
    No Mapping
    No Content Based Routing
    No Content Conversion
    No Custom Modules
    Probably you are doing content conversion that is why it is not working.
    Hope this helps,
    Mark
    Edited by: Mark Dihiansan on Mar 5, 2012 12:58 PM

  • How to read HTML files using UTL_FILE

    Hello Friends,
    How to read HTML files using UTL_FILE package ? According
    to Oracle documentation UTL_FILE can read or write OS Text Files.
    Thanx in advance..
    Adi

    HI Hareesh,
    i have gone through that blog.
    i tried it...but i am getting mapping error  no receiver determination fond because there are so  many excel files.
    my data is available on sharedString.xml but also it is in not same order.
    i have no clue how to handle this part form the blog.
    "This way our mapping will receive all data from the sheet in an XML format. The only thing that's left is to create an XSD file from the XML file we received in order to be able to use it in the mapping and as our Service Interface and we can proceed with mapping. As you can see from the sheet.xml files all the data is placed with column name and row number so it's not that difficult to map it to an table type format using the Message Mapping only (no java, abap mapping required)."

  • Is it possible to read the file using File Adapter which is in client machi

    Is it possible to read the file using file Adapter which is in client machine(on the same network).Then what is thee need of FTP Adapter?

    You can achieve that by exposing your client machine as a drive, then you can go using file adapter.
    FTP adapter will help you to communicate with different machine which is in different network.
    It is considered good etiquette to reward answerers with points (as "helpful" - 5 pts - or "correct" - 10pts).
    Thanks,
    Vijay

  • How to read pdf file using file adapter

    Hi..
        How to read pdf file using file adapter?
    regards
    Arun

    Hi
    This may help you
    /people/sap.user72/blog/2005/07/27/xi-generate-pdf-file-out-of-file-adapter
    /people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover
    ---Ram

  • How to read 835 files using ssis

    Hello Everyone,
    It is possible read 835 files using ssis.
    Please share your suggestions on this.
    Regards,
    Vaishu

    Hi Vaishu,
    None of the canned (standard) SSIS tasks let you do so.
    But if you can buy http://www.cozyroc.com/ssis/edi-source (I am not anyhow affiliated with CozyRoc) it states it can read ERA (AKA 835) files. CozyRoc lets you run it for free in Dev.
    If there will be a decision not to buy then you can simply crate a transformation out of several manipulations or by creating a custom SSIS component or some other programming methods need to be used.
    PS: Again, I just know CozyRoc quality of components is awesome, but I do not make any profit from referring to it.
    Arthur My Blog

  • I am unable to read pdf files using Acrobat Reader on my MAC OS X. Any suggestions?

    How do I read pdf files using my MAC OS X?

    You can read pdf files with preview. Select a pdf document, right click on it, and choose Preview. You can also get info and select to choose Preview for all.
    You can also install Adobe Reader 10.1.3 for Lion. It will also install a plug-in to read pdf docs in the browser.
    http://www.adobe.com/support/downloads/detail.jsp?ftpID=5360

  • How to create and read text file using LabVIEW 7.1 PDA module?

    How to create and read text file using LabVIEW 7.1 PDA module? I can not create a text file and read it.
    I attach my code here.
    Attachments:
    File_IO.vi ‏82 KB

    Well my acquisition code runs perfect. The problem is reading it. I can't seem to read my data no matter what I do. My data gets saved as a string using the array to string vi but I've read that the string to array vi (which I need to convert back to array to read my data) does not work on the pda. I'm using version 8.0. So I was trying to modify the program posted in this discussion so that it would save data from my DAQ. I did that but I still can't read the data after its saved. I really don't know what else to do. All I need to do is read the data on the pda itself. I can't understand why I'm having such a hard time doing that. I found a possible solution on another discussion that talks about parsing the strings because of the bug in the "string to array" vi. However, that lead me to another problem because for some reason, the array indicators or graphs don't function on the pda. When i build the program to the pda or emulator, the array indicators are faded out on the front panel as if the function is not valid. Does this kind of help give a better picture of what I'm trying to do. Simply read data back. Thanks.

  • Elements 9 failure to read large files.

    I recently purchased a Nikon D5300. I was a previous D50 user and always saved and converted RAW NEF files with Elements 9. The D5300 save 24Mb NEF files and Elements 9 would not read these files. I chanded to save RAW and JPEG files and it would not read those either. It will read lower res JPEG files. Is there an update to Elements 9 to alllow reading larger files?
    Bob Jacobson
    [email protected]

    Here's an interesting take on the "dependent operation failure" error message
    Adobe CS5: Payload cannot be installed due to dependent operation failure

Maybe you are looking for