Reading large files -- use FileChannel or BufferedReader?
Question --
I need to read files and get their content. The issue is that I have no idea how big the files will be. My best guess is that most are less than 5kb but some with be huge.
I have it set up using a BufferedReader, which is working fine. It's not the fastest thing (using readLine() and StringBuffer.append()), but so far it's usable. However, I'm worried that if I need to deal with large files, such as a PDF or other binary, BufferedReader won't be so efficient if I do it line by line. (And will I run into issues trying to put a binary file into a String?)
I found a post that recommended FileChannel and ByteBuffer, but I'm running into a java.lang.UnsupportedOperationException when trying to get the byte[] from ByteBuffer.
File f = new File(binFileName);
FileInputStream fis = new FileInputStream(f);
FileChannel fc = fis.getChannel();
// Get the file's size and then map it into memory
int sz = (int)fc.size();
MappedByteBuffer bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, sz);
fc.close();
String contents = new String(bb.array()); //code blows up
Thanks in advance.
If all you are doing is reading data I don't think you're going to get much faster than InfoFetcher
you are welcome to use and modify this class, but please don't change the package or take credit for it as your own work
InfoFetcher.java
==============
package tjacobs.io;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Iterator;
* InfoFetcher is a generic way to read data from an input stream (file, socket, etc)
* InfoFetcher can be set up with a thread so that it reads from an input stream
* and report to registered listeners as it gets
* more information. This vastly simplifies the process of always re-writing
* the same code for reading from an input stream.
* <p>
* I use this all over
public class InfoFetcher implements Runnable {
public byte[] buf;
public InputStream in;
public int waitTime;
private ArrayList mListeners;
public int got = 0;
protected boolean mClearBufferFlag = false;
public InfoFetcher(InputStream in, byte[] buf, int waitTime) {
this.buf = buf;
this.in = in;
this.waitTime = waitTime;
public void addInputStreamListener(InputStreamListener fll) {
if (mListeners == null) {
mListeners = new ArrayList(2);
if (!mListeners.contains(fll)) {
mListeners.add(fll);
public void removeInputStreamListener(InputStreamListener fll) {
if (mListeners == null) {
return;
mListeners.remove(fll);
public byte[] readCompletely() {
run();
return buf;
public int got() {
return got;
public void run() {
if (waitTime > 0) {
TimeOut to = new TimeOut(waitTime);
Thread t = new Thread(to);
t.start();
int b;
try {
while ((b = in.read()) != -1) {
if (got + 1 > buf.length) {
buf = IOUtils.expandBuf(buf);
int start = got;
buf[got++] = (byte) b;
int available = in.available();
//System.out.println("got = " + got + " available = " + available + " buf.length = " + buf.length);
if (got + available > buf.length) {
buf = IOUtils.expandBuf(buf, Math.max(got + available, buf.length * 2));
got += in.read(buf, got, available);
signalListeners(false, start);
if (mClearBufferFlag) {
mClearBufferFlag = false;
got = 0;
} catch (IOException iox) {
throw new PartialReadException(got, buf.length);
} finally {
buf = IOUtils.trimBuf(buf, got);
signalListeners(true);
private void setClearBufferFlag(boolean status) {
mClearBufferFlag = status;
public void clearBuffer() {
setClearBufferFlag(true);
private void signalListeners(boolean over) {
signalListeners (over, 0);
private void signalListeners(boolean over, int start) {
if (mListeners != null) {
Iterator i = mListeners.iterator();
InputStreamEvent ev = new InputStreamEvent(got, buf, start);
//System.out.println("got: " + got + " buf = " + new String(buf, 0, 20));
while (i.hasNext()) {
InputStreamListener fll = (InputStreamListener) i.next();
if (over) {
fll.gotAll(ev);
} else {
fll.gotMore(ev);
}
Similar Messages
-
Read Large Files Using BPEL File Adapter
Hi,
I have a scenario, where files of large size in text format are to be read and send to 3rd party. MTOM Policy has to be attached. Files of size 3MB or less are polled and size greater than 3 MB are not Retrieved. How can I resolve the issue. Do I have to break the file and send the data? If so how?
Thanks
RangaYou have to use streaming feature of the JCA file adapter for handling huge file.
You could go through the following link
http://docs.oracle.com/cd/E23943_01/integration.1111/e10231/adptr_file.htm#CIAHHEBF -
File Adapter Not Reading Large Files
Dear Experts,
Enviornment :-
OS:-Linux
Jdeveloper:- 11.1.1.6
SOA:-11.1.1.6
Weblogic:-10.3.6
JDK:-SUN
Allocated RAM:-16GB
Currently we are in UAT Phase and now we are facingan issue in reading large file.Below is the Design details of the service
FileAdapter(Read)-->Bpel(Business login,Using FlowN)-->FileAdapter(Write CSV),JMS Adapter(AQ JMS topics)
In this case at the time of read itself we are facing issue. File adapter reading xml file but in receive activity receives input data as
xmlDocKey:1C135990067411E3BFA6B5087B629F9DI
I really couldn;t understand about the error. Even i tried reading as Opaque format and still end up with same error.
In order to make sure i have create mediator and tried reading the file, in case i was able to read file upto 15mb with out any error.Also i tried as "read as Attachment" in Bpel component and able to read the attachment upto 7 mb file, but this is hitting the performance side.
I request some one please let me know why the file adapter is giving XmlDOCKey rather then XMLContent to the inputvariable
Regards,
TarakCan you check your BPEL Properties in EM?
Go to Soa-infra > right Click > SOA Administration > BPEL properties
increase the Dispatcher Engine Thread = 10, Dispatcher Invoke Threads = 60 and Dispatcher Engine Threads to 90
Click on "More BPEL Configuration Properties"
Increase the DispatcherMaxRequestDepth from 600 to 1000.
Bounce the server and see if works..
Bounce the server and try again.
If this fails, try get a threshold by increasing the file size until it fails again. -
Reading large file with JCA Adapter in OSB
Hello,
We are searching for a solution how to read large file (>50M) from network drive and deliver it to queue via OSB 11gR4 (10.3.4). The problem is when reading the file with JCA File Adapter. It seems that it cannot handle as large files as we have. The documentation provides a way to bypass file size limitation by using Chunk Read but it seems to require BPEL Process execution which is not possible in our environment. Does anyone know if there are ways to implement this without having BPEL Process?
Our usecase:
read file from network drive -> transfer with OSB -> deliver MQ
Other options than JCA File Adapter can be considered, if anyone can advice...If it's a plain routing use case and no message processing is required then you may simply use OSB's FILE transport instead of JCA adapter. Create a messaging type proxy service and select request message type as "binary". Also enable the content streaming (Disk buffer, compression).
From OSB Dev guide -
Oracle JCA Adapter for FTP and Files – Attachments (large payload support), pre- and post-processing of files, using a re-entrant valve for processing ZIP files, content streaming, and file chunked read are not supported with Oracle Service Bus.
http://download.oracle.com/docs/cd/E17904_01/doc.1111/e15866/jca.htm#BABBICIA
You may also refer -
Reading huge flat file in OSB 11gR1
Regards,
Anuj -
How to read pdf files using java.io package classes
Dear All,
I have a certain requirement that i should read and write PDF files at runtime. With normal java file IO reading is not working. Can any one suggest me how to proceed probably with sample code block
Thanks in advance.hi I also have the pbm. to read pdf file using JAVA
can any body help meWhy is it so difficult to read the thread you posted in? They say: java.io is pointless, use iText. So why don't you?
or also I want to read a binary encoded data into
ascii,
can anybody give me a hint how to do it.Depends on what you mean with "binary encoding". ASCII's binary encoding, too, basically. -
Reading XML file using BAPI and then uploading that xml file data into SAP
I am getting a xml file from Java server. I need to take
data from this file using BAPI and need to upload into SAP using SAP.
Please tell me how to read XML files using BAPI's.<b>SDIXML_DATA_TO_DOM</b> Convert SAP data (elementary/structured/table types) into DOM (XML
<b>SDIXML_DOM_TO_XML</b> Convert DOM (XML) into string of bytes that can be downloaded to PC or application server
<b>SDIXML_DOM_TO_SCREEN</b> Display DOM (XML)
<b>SDIXML_DOM_TO_DATA</b>
data: it_table like t001 occurs 0.
data: l_dom TYPE REF TO IF_IXML_ELEMENT,
m_document TYPE REF TO IF_IXML_DOCUMENT,
g_ixml TYPE REF TO IF_IXML,
w_string TYPE XSTRING,
w_size TYPE I,
w_result TYPE I,
w_line TYPE STRING,
it_xml TYPE DCXMLLINES,
s_xml like line of it_xml,
w_rc like sy-subrc.
start-of-selection.
select * from t001 into table it_table.
end-of-selection.
initialize iXML-Framework ****
write: / 'initialiazing iXML:'.
class cl_ixml definition load.
g_ixml = cl_ixml=>create( ).
check not g_ixml is initial.
write: 'ok'.
create DOM from SAP data ****
write: / 'creating iXML doc:'.
m_document = g_ixml->create_document( ).
check not m_document is initial.
write: 'ok'.
write: / 'converting DATA TO DOM 1:'.
CALL FUNCTION 'SDIXML_DATA_TO_DOM'
EXPORTING
NAME = 'IT_TABLE'
DATAOBJECT = it_table[]
IMPORTING
DATA_AS_DOM = l_dom
CHANGING
DOCUMENT = m_document
EXCEPTIONS
ILLEGAL_NAME = 1
OTHERS = 2.
if sy-subrc = 0. write 'ok'.
else. write: 'Err =', sy-subrc.
endif.
check not l_dom is initial.
write: / 'appending DOM to iXML doc:'.
w_rc = m_document->append_child( new_child = l_dom ).
if w_rc is initial. write 'ok'.
else. write: 'Err =', w_rc.
endif.
visualize iXML (DOM) ****
write: / 'displaying DOM:'.
CALL FUNCTION 'SDIXML_DOM_TO_SCREEN'
EXPORTING
DOCUMENT = m_document
EXCEPTIONS
NO_DOCUMENT = 1
OTHERS = 2.
if sy-subrc = 0. write 'ok'.
else. write: 'Err =', sy-subrc.
endif.
convert DOM to XML doc (table) ****
write: / 'converting DOM TO XML:'.
CALL FUNCTION 'SDIXML_DOM_TO_XML'
EXPORTING
DOCUMENT = m_document
PRETTY_PRINT = ' '
IMPORTING
XML_AS_STRING = w_string
SIZE = w_size
TABLES
XML_AS_TABLE = it_xml
EXCEPTIONS
NO_DOCUMENT = 1
OTHERS = 2.
if sy-subrc = 0. write 'ok'.
else. write: 'Err =', sy-subrc.
endif.
write: / 'XML as string of size:', w_size, / w_string.
describe table it_xml lines w_result.
write: / 'XML as table of', w_result, 'lines:'..
loop at it_xml into s_xml.
write s_xml.
endloop.
write: / 'end of processing'.
end of code
Hope this will be useful.
regards
vinod -
Reading XML file using BAPI I must use adapters .
Reading XML file using BAPI and then uploading that xml file data into SAP using BDC.
I cant take file on to my Application server I am getting the file dynamically from other file server and I need to use BAPis to read data from XML file.please tell me what should be my Import,Export and Tables parameterrs should be.
ThanksHi,
Import, export and table parameters for BAPI is required, without that BAPI will not able to collect the data from XML. What you need to do is write a Zprogram and collect the data, store that data in internal table and call the BAPI by passing required parameters.
Different Scanarios:
1) Before calling a BAPI write some other program which collects the data from XML and create a UNIX file. Try to get the data from UNIX FILE
2) If you see the XML tags, data will be in side that tags, try to get the data from XML tags and store that data in one SAP table. You can use the BAPI by fetching the data from table
3)Create a table and store the data in the table. Use the table in SAP to extract the data.
BAPI won't work without any parameters, you have to pass some parameters then the BAPI will return some values.
Hope i am clear.
Thanks&Regards,
-Suresh Revuru -
Processing Large Files using Chunk Mode with ICO
Hi All,
I am trying to process Large files using ICO. I am on PI 7.3 and I am using new feature of PI 7.3, to split the input file into chunks.
And I know that we can not use mapping while using Chunk Mode.
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
Thanks in Advance,
- Pooja.Hello,
While trying I noticed below points:
1) I had Created Data Type, Message Type and Interfces in ESR and used the same in my scenario (No mapping was defined)Sender and receiver DT were same.
Result: Scenario did not work. It created only one Chunk file (.tmp file) and terminated.
2) I used Dummy Interface in my scenario and it worked Fine.
So, Please confirm if we should always USE DUMMY Interfaces in Scenario while using Chunk mode in PI 7.3 Or Is there something that I am missing.
According to this blog:
File/FTP Adapter - Large File Transfer (Chunk Mode)
The following limitations apply to the chunk mode in File Adapter
As per the above screenshots, the split never cosiders the payload. It's just a binary split. So the following limitations would apply
Only for File Sender to File Receiver
No Mapping
No Content Based Routing
No Content Conversion
No Custom Modules
Probably you are doing content conversion that is why it is not working.
Hope this helps,
Mark
Edited by: Mark Dihiansan on Mar 5, 2012 12:58 PM -
How to read HTML files using UTL_FILE
Hello Friends,
How to read HTML files using UTL_FILE package ? According
to Oracle documentation UTL_FILE can read or write OS Text Files.
Thanx in advance..
AdiHI Hareesh,
i have gone through that blog.
i tried it...but i am getting mapping error no receiver determination fond because there are so many excel files.
my data is available on sharedString.xml but also it is in not same order.
i have no clue how to handle this part form the blog.
"This way our mapping will receive all data from the sheet in an XML format. The only thing that's left is to create an XSD file from the XML file we received in order to be able to use it in the mapping and as our Service Interface and we can proceed with mapping. As you can see from the sheet.xml files all the data is placed with column name and row number so it's not that difficult to map it to an table type format using the Message Mapping only (no java, abap mapping required)." -
Is it possible to read the file using File Adapter which is in client machi
Is it possible to read the file using file Adapter which is in client machine(on the same network).Then what is thee need of FTP Adapter?
You can achieve that by exposing your client machine as a drive, then you can go using file adapter.
FTP adapter will help you to communicate with different machine which is in different network.
It is considered good etiquette to reward answerers with points (as "helpful" - 5 pts - or "correct" - 10pts).
Thanks,
Vijay -
How to read pdf file using file adapter
Hi..
How to read pdf file using file adapter?
regards
ArunHi
This may help you
/people/sap.user72/blog/2005/07/27/xi-generate-pdf-file-out-of-file-adapter
/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover
---Ram -
How to read 835 files using ssis
Hello Everyone,
It is possible read 835 files using ssis.
Please share your suggestions on this.
Regards,
VaishuHi Vaishu,
None of the canned (standard) SSIS tasks let you do so.
But if you can buy http://www.cozyroc.com/ssis/edi-source (I am not anyhow affiliated with CozyRoc) it states it can read ERA (AKA 835) files. CozyRoc lets you run it for free in Dev.
If there will be a decision not to buy then you can simply crate a transformation out of several manipulations or by creating a custom SSIS component or some other programming methods need to be used.
PS: Again, I just know CozyRoc quality of components is awesome, but I do not make any profit from referring to it.
Arthur My Blog -
I am unable to read pdf files using Acrobat Reader on my MAC OS X. Any suggestions?
How do I read pdf files using my MAC OS X?
You can read pdf files with preview. Select a pdf document, right click on it, and choose Preview. You can also get info and select to choose Preview for all.
You can also install Adobe Reader 10.1.3 for Lion. It will also install a plug-in to read pdf docs in the browser.
http://www.adobe.com/support/downloads/detail.jsp?ftpID=5360 -
How to create and read text file using LabVIEW 7.1 PDA module?
How to create and read text file using LabVIEW 7.1 PDA module? I can not create a text file and read it.
I attach my code here.
Attachments:
File_IO.vi 82 KBWell my acquisition code runs perfect. The problem is reading it. I can't seem to read my data no matter what I do. My data gets saved as a string using the array to string vi but I've read that the string to array vi (which I need to convert back to array to read my data) does not work on the pda. I'm using version 8.0. So I was trying to modify the program posted in this discussion so that it would save data from my DAQ. I did that but I still can't read the data after its saved. I really don't know what else to do. All I need to do is read the data on the pda itself. I can't understand why I'm having such a hard time doing that. I found a possible solution on another discussion that talks about parsing the strings because of the bug in the "string to array" vi. However, that lead me to another problem because for some reason, the array indicators or graphs don't function on the pda. When i build the program to the pda or emulator, the array indicators are faded out on the front panel as if the function is not valid. Does this kind of help give a better picture of what I'm trying to do. Simply read data back. Thanks.
-
Elements 9 failure to read large files.
I recently purchased a Nikon D5300. I was a previous D50 user and always saved and converted RAW NEF files with Elements 9. The D5300 save 24Mb NEF files and Elements 9 would not read these files. I chanded to save RAW and JPEG files and it would not read those either. It will read lower res JPEG files. Is there an update to Elements 9 to alllow reading larger files?
Bob Jacobson
[email protected]Here's an interesting take on the "dependent operation failure" error message
Adobe CS5: Payload cannot be installed due to dependent operation failure
Maybe you are looking for
-
Catalog id is not getting displayed properly in Foreign Language (French)
Hi All, I have created a web service id for the external catalog and I have assigned the same to one of the user. The name of the web service id is getting displayed properly in the shopping cart creation screen when the language is English. But when
-
hi, i have a nokia C3-00 need unloking code; can you people help me Moderator's note: We have removed personal information from your post because we do not recommend you sharing this on a public discussions board.
-
Applet transfer from one card to another card
Can any body tell me is it possibel to transfer card applet from one java card to another java card
-
I have looked at the settings, they seem ok...I dont get the bar at the top asking if I would like FFox to remember passwords. It was working earlier this year but not now....not sure if it since FFox was updated
-
Setting Background Colour with a Value Retrieved from a Database
I'm retrieving different colours from a database and I want to set the background colour of a canvas to the colour retrieved. I've tried doing the following: <mx:TileList x="2" y="237" direction="horizontal" dataProvider="{MakeUpColours.lastResult.Co