Unable to read big files into string object & java.lang.OutOfMemory Problem
Hi All,
I have an application that uses applet and servlet communication. On the client side I am reading an large xml file of 12MB size (using JFileChooser) and converting the file to an string object using below code. But I am getting java.lang.OutOfMemory on the client side . But the same below code works fine for small xml files which are less than 4MB sizes:
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(file),"UTF8"), 1024*12);
String s, s2 = new String();
while((s = in.readLine())!= null)
s2 += s + "\n";
I even tried the below code but still java.lang.OutOfMemory is coming:
while (true)
int i = in.read();
if (i == -1)
break;
sb.append(i);
Please let me know what am I doing wrong here ...
Hi,
I could avoid the java.lang.OutOfMemory error using below code. But using below code I could read small files of sizes less than 4MB
but with large files of 12 MB the below code just simply hangs and I am unable to print the string object namely 's'.
My purpose is to construct an String or StringBuffer object out the user uploaded xml file at the client side and pass that object to server for processing. So how can I construct such object avoid memory problem and increasing the performance of such operations.
BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
byte[] b = new byte[in.available()];
in.read(b, 0, b.length);
String s = new String(b, 0, b.length);
in.close();
Thanks & Regards,
Sony.
Similar Messages
-
Unable to read big files into string object and java.lang.OutOfMemory Prob
Hi All,
I have an application that uses applet and servlet communication. On the client side I am reading an large xml file of 12MB size (using JFileChooser) and converting the file to an string object using below code. But I am getting java.lang.OutOfMemory on the client side . But the same below code works fine for small xml files which are less than 4MB sizes:
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(file),"UTF8"), 1024*12);
String s, s2 = new String();
while((s = in.readLine())!= null)
s2 += s + "\n";
I even tried below code but still java.lang.OutOfMemory is coming:
while (true)
int i = in.read();
if (i == -1)
break;
sb.append(i);
Please let me know what am I doing wrong here ...
Thanks & Regards,
Sony.Using a String is bad for the following reason:
When you initially create the String, it has a certain memory size (allocated length if you will). As you keep appending to this String, then memory reallocation will occur over and over, slowing your program down dramatically (ive seen with a 16k x 8 Char file taking 30 secs to read into memory using Strings in this way)
A Better way would be if you knew the number of characters in the XML file (Using some File size method for example) Then you can use a StringBuffer, which will pre allocate enough space (or try to, it may just be that you cannot create a string as large as you need). You can use toString() method to get the resultant in a String Object (the extra allocated space at the end of the Buffer will be removed)
StringBuffer strBuf = new StringBuffer(xxxx);
Where xx is the length (int). Assuming that you are only allowed to enter an int to the constructor then (platform depedant) an int is 2^31 at maximum (or whatever) which allows 2.14e9 characters, therefore an xml file being totally filed would allow a size of ~2048 MB to be read in.
Try it and see. -
Unable to Read XML files into PI 7.0
Dear Friends,
I am working on File to IDOC and IDoc to File scenario with a shared folder concept.
we have tested all our transactions to and fro with out any problem. Two days back we faced sudden server shut down due to power issue.
From that day we are facing problem in reading files into XI
We are not facing any problem during outbound transactions from SAP on the same business system, where inbound folder path is also mentioned.
I have checked in Monitoring under communication channel..
it is showing comm channal is correctly configured...
is there any way i can check where it is going wrong..
Any thoughts and suggestions is highly rewarded.
Thanks & Regards,
SumanHi Santhosh,
We have given full folder permissions for all users. Processed read file option is not enabled in my Sender CC.
Thanks for quick response.
Regards,
Suman -
OutOfMemoryError while attempting to read 60MB file into HashMap objects
Hi,
I've a 60MB file with about 60000 records that I'm trying to read into a HashMap of <String, String[]>. However, as far as I can see it can only manage reading in 12000 before it falls over with an OutOfMemoryError.
My eclipse.ini contains the following entries:
-Xms40m
-Xmx512m
I've changed Xmx512m to Xmx1024m but this hasn't made a difference.
My program involves reading in the file, sorting it based on 2 attributes of a record and then outputting it in the sorted order.
I've tried a few other methods around just reading in the attributes that I need, doing the sort with those, looking for the rows I need from the input file sequentially and then outputting this in the correct order but this seems to be taking forever, well over 12 hours!
How else can I go about this? Thanks.Here is the code. Sorry, how do I go about compiling this class as it's looking for the CsvReader jar but can't find. Have tried the -cp option but doesn't seems to work. Thanks.
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.HashMap;
import com.csvreader.CsvReader;
public class Reader {
* @param args
* @throws IOException
public static void main(String[] args) throws IOException {
CsvReader reader = new CsvReader(new FileReader("C:\\temp\\gen_ppl.csv"), '|');
HashMap<String, String[]> allCorpIDs = new HashMap<String, String[]>();
while(reader.readRecord())
allCorpIDs.put(reader.get(0), reader.getValues());
} -
Convert a line read from text file into string
how to convert a line from text file into string?
i know how to convert to numbers,
private int parseLine1(String line) {
StringTokenizer tokenizer = new StringTokenizer(line);
value1 = Integer.valueOf(tokenizer.nextToken()).intValue();
value2 = Integer.valueOf(tokenizer.nextToken()).intValue();
return value1;
}but what about charactrs?ok, here is my problem, i have a file with a bunch of Xs in it but position function doesn't return a correct value, what's going wrong?
private int positioni(){
int i=0;
int j=0;
int b=0;
String line="";
while(line!= null){
for(int a=0; a<line.length(); a++){
if(line.charAt(a)=='X'){
i=a;}
b++;
j=b;
t=line.length();
line=read(gridFileN);
return i;
private String read(String ggridFileN){
TextStreamReader ggridFile = new TextStreamReader(ggridFileN);
return ggridFile.readLine(); -
Reading in text file into strings
Hi,
I'm new to the reading in data part of java and I can't work out a good way to read in a .txt file and then store each line of the txt file into strings.
I think I'm probably just missing something obvious but if anyone can help :D!
Thanks,
BexLook up Scanner in Java 1.5, or BufferedReader in Java 1.4.
-
Set encoding to read a file into a string
in my Coding i Want to read a file into a string like this
File myFile = new File (Dateipfad);
System.out.println("folgende Datei wird nun verarbeitet: " + Dateipfad);
java.io.BufferedReader myBis = new java.io.BufferedReader(new java.io.FileReader(myFile.getAbsolutePath()));
myBis.StringBuffer buf = new StringBuffer();
String line = myBis.readLine();
while (line != null)
buf.append(line);
line = myBis.readLine();
myBis.close();
] now i just want to set an encoding "ISO-8859-1", so that all �`s, �`s or �`s dont get lost.
Can Anyone help me please ?!You must use java.io.InputStreamReader to set the required encoding.
See the API for more details.
http://java.sun.com/j2se/1.3/docs/api/java/io/InputStreamReader.html
HTH,
Manuel Amago. -
How to read a binaryfile into an object
Hello,
I want to read a binary file into an object. In this object, we have
several variables which need to be filled. I did the following thing:
---Start code---
import java.io.*;
import java.lang.reflect.*;
public class BinaryFile {
private File file;
private DataInputStream dis;
public BinaryFile() {}
// Open the file
public void openFile(String file) throws FileNotFoundException {
this.file = new File(file);
if (! this.file.exists())
throw new FileNotFoundException();
dis = new DataInputStream(new FileInputStream(this.file));
// Close the file
public void closeFile() throws IOException {
dis.close();
public void readObject(int offset, Object o) throws EndOfFileException,
IOException {
// Get the fields in the object
Field[] fields = o.getClass().getFields();
try {
String type, naam;
Class className;
int lengte, arrayLengte;
boolean isArray;
for (int i = 0; i !< fields.length; i++) {
naam = fields.getName(); // Get the name of the field
className = fields[i].getType(); // Get the type
type = className.getName(); // Get the name of the type
isArray = className.isArray(); // Is it an array?
arrayLengte = 1;
fields[i].setAccessible(true); // For setting the values
if (isArray) { // If it is an array,
type = getType(type); // decode it
// Get the size of the array
arrayLengte = Array.getLength(fields[i].get(o));
// Determine the type of the variable and fill it.
// Char
if (type.equalsIgnoreCase("char")) {
lengte = 1;
if (isArray)
fields[i].set(o, readChars(arrayLengte * lengte));
else
fields[i].setChar(o, readChar());
// Byte
} else if (type.equalsIgnoreCase("byte")) {
lengte = 1;
if (isArray)
fields[i].set(o, readBytes(arrayLengte * lengte));
else
fields[i].setByte(o, readByte());
// Short
} else if (type.equalsIgnoreCase("short")) {
lengte = 2;
if (isArray)
fields[i].set(o, readShorts(arrayLengte * lengte));
else
fields[i].setShort(o, readShort());
// Int
} else if (type.equalsIgnoreCase("int")) {
lengte = 4;
if (isArray)
fields[i].set(o, readInts(arrayLengte * lengte));
else
fields[i].setInt(o, readInt());
// Float
} else if (type.equalsIgnoreCase("float")) {
lengte = 4;
if (isArray)
fields[i].set(o, readFloats(arrayLengte * lengte));
else
fields[i].setFloat(o, readFloat());
// Long
} else if (type.equalsIgnoreCase("long")) {
lengte = 8;
if (isArray)
fields[i].set(o, readLongs(arrayLengte * lengte));
else
fields[i].setLong(o, readLong());
// Double
} else if (type.equalsIgnoreCase("double")) {
lengte = 8;
if (isArray)
fields[i].set(o, readDoubles(arrayLengte * lengte));
else
fields[i].setDouble(o, readDouble());
catch (IOException ioex) {
throw ioex;
catch (Exception ex) {
ex.printStackTrace();
// Methods for reading from the DataInputStream
private char[] readChars(int lengte) throws IOException {
char[] c = new char[lengte];
for (int i = 0; i < c.length; i++)
c[i] = dis.readChar();
return c;
private char readChar() throws IOException {
return dis.readChar();
private byte[] readBytes(int lengte) throws IOException {
byte[] b = new byte[lengte];
for (int i = 0; i < b.length; i++)
b[i] = dis.readByte();
return b;
private byte readByte() throws IOException {
return dis.readByte();
private short[] readShorts(int lengte) throws IOException {
short[] s = new short[lengte];
for (int i = 0; i < s.length; i++)
s[i] = dis.readShort();
return s;
private short readShort() throws IOException {
return dis.readShort();
private int[] readInts(int lengte) throws IOException {
int[] i = new int[lengte];
for (int j = 0; j < i.length; j++)
i[j] = dis.readInt();
return i;
private int readInt() throws IOException {
return dis.readInt();
private double[] readDoubles(int lengte) throws IOException {
double[] d = new double[lengte];
for (int i = 0; i < d.length; i++)
d[i] = dis.readDouble();
return d;
private double readDouble() throws IOException {
return dis.readDouble();
private float[] readFloats(int lengte) throws IOException {
float[] f = new float[lengte];
for (int i = 0; i < f.length; i++)
f[i] = dis.readFloat();
return f;
private float readFloat() throws IOException {
return dis.readFloat();
private long[] readLongs(int lengte) throws IOException {
long[] l = new long[lengte];
for (int i = 0; i < l.length; i++)
l[i] = dis.readLong();
return l;
private long readLong() throws IOException {
return dis.readLong();
// If it is an array, decode it. The type will be "[x" where x is
// translated according to the table
private String getType(String type) {
BaseType Character Type Interpretation
B byte signed byte
C char Unicode character
D double double-precision floating-point value
F float single-precision floating-point value
I int integer
J long long integer
L<classname>; reference an instance of class <classname>
S short signed short
Z boolean true or false
[ reference one array dimension
int arrayIndex = type.indexOf("[");
if (arrayIndex != type.lastIndexOf("[")) {
System.out.println("Multiple arrays not supported!");
return type;
char tempType = type.charAt(arrayIndex + 1);
switch (tempType) {
case 'B':
type = "byte";
break;
case 'C':
type = "char";
break;
case 'D':
type = "double";
break;
case 'F':
type = "float";
break;
case 'I':
type = "int";
break;
case 'J':
type = "long";
break;
case 'S':
type = "short";
break;
default:
System.out.println("Couldn't define this type: " + type);
type = type;
break;
return type;
---End code---
Now the problem. I want to read the header of an executable. I turn to
the file format as described in WINNT.H. If it is a char, it will read
ok. For everything else, it doesn't work. A WORD (= short in java) for
instance, doesn't come up with the real value.
The purpose of this piece of code is to emulate the structs that are
used in C.
What is the problem here and what is the solution?before coding a long class, look at Serialization mechanism in java
maybe (or maybe not) it should be a better soution ... -
Hello,
I want to convert total xml file into string in UDF, i am getting an error. can anybody tell me the solution. what is the wrong in bellow code.
String str =a+b;
String strLine;
StringBuffer sbr=new StringBuffer();
try{
FileInputStream fileInputStream = new FileInputStream(str);
DataInputStream dataInputStream = new DataInputStream(fileInputStream);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(dataInputStream));
while ((strLine = bufferedReader.readLine()) != null) {
sbr.append(strLine);
catch (Exception e){ e.printStackTrace(); }
return sbr.toString();}check this link
How to store big XML string in ABAB DB
regards,
srinivas
<b>*reward for useful answers*</b> -
Unable to read payload from the message object in XI
Hello Guys,
Please help me about my problem in XI version 7.0.im quite new here.
im trying to test my config but error message occured. "Unable to read payload from the message object"
when i checked the comm channel this is the error message :
Error during database connection to the database URL 'jdbc:sqlserver://172.16.40.20:1433;databasename=TRAVEL:SelectMethod=cursor' using the JDBC driver 'com.microsoft.sqlserver.jdbc.SQLServerDriver': 'com.sap.aii.adapter.jdbc.sql.DriverManagerException: Cannot establish connection to URL 'jdbc:sqlserver://172.16.40.20:1433;databasename=TRAVEL:SelectMethod=cursor': com.microsoft.sqlserver.jdbc.SQLServerException: Cannot open database "TRAVEL:SelectMethod=cursor" requested by the login. The login failed.'
when i tried my login in sql it works...but in this message the login is failed..what shall i do..
Please advice.
Thanks in advance
aVaDuDzHi
Check with the connection string & Authorization of user you have used.
MSSQL string is
jdbc:microsoft:sqlserver://dbhost:1433;databaseName=example;SelectMethod=Cursor
While doing JDBC its good to refer Note 831162 lot of problems can be resolved.
Thanks
Gaurav -
Question about reading csv file into internal table
Some one (thanks those nice guys!) in this forum have suggested me to use FM KCD_CSV_FILE_TO_INTERN_CONVERT to read csv file into internal table. However, it can be used to read a local file only.
I would like to ask how can I read a CSV file into internal table from files in application server?
I can't simply use SPLIT as there may be comma in the content. e.g.
"abc","aaa,ab",10,"bbc"
My expected output:
abc
aaa,ab
10
bbb
Thanks again for your help.Hi Gundam,
Try this code. I have made a custom parser to read the details in the record and split them accordingly. I have also tested them with your provided test cases and it work fine.
OPEN DATASET dsn FOR input IN TEXT MODE ENCODING DEFAULT.
DO.
READ DATASET dsn INTO record.
PERFORM parser USING record.
ENDDO.
*DATA str(32) VALUE '"abc",10,"aaa,ab","bbc"'.
*DATA str(32) VALUE '"abc","aaa,ab",10,"bbc"'.
*DATA str(32) VALUE '"a,bc","aaaab",10,"bbc"'.
*DATA str(32) VALUE '"abc","aaa,ab",10,"b,bc"'.
*DATA str(32) VALUE '"abc","aaaab",10,"bbc"'.
FORM parser USING str.
DATA field(12).
DATA field1(12).
DATA field2(12).
DATA field3(12).
DATA field4(12).
DATA cnt TYPE i.
DATA len TYPE i.
DATA temp TYPE i.
DATA start TYPE i.
DATA quote TYPE i.
DATA rec_cnt TYPE i.
len = strlen( str ).
cnt = 0.
temp = 0.
rec_cnt = 0.
DO.
* Start at the beginning
IF start EQ 0.
"string just ENDED start new one.
start = 1.
quote = 0.
CLEAR field.
ENDIF.
IF str+cnt(1) EQ '"'. "Check for qoutes
"CHECK IF quotes is already set
IF quote = 1.
"Already quotes set
"Start new field
start = 0.
quote = 0.
CONCATENATE field '"' INTO field.
IF field IS NOT INITIAL.
rec_cnt = rec_cnt + 1.
CONDENSE field.
IF rec_cnt EQ 1.
field1 = field.
ELSEIF rec_cnt EQ 2.
field2 = field.
ELSEIF rec_cnt EQ 3.
field3 = field.
ELSEIF rec_cnt EQ 4.
field4 = field.
ENDIF.
ENDIF.
* WRITE field.
ELSE.
"This is the start of quotes
quote = 1.
ENDIF.
ENDIF.
IF str+cnt(1) EQ ','. "Check end of field
IF quote EQ 0. "This is not inside quote end of field
start = 0.
quote = 0.
CONDENSE field.
* WRITE field.
IF field IS NOT INITIAL.
rec_cnt = rec_cnt + 1.
IF rec_cnt EQ 1.
field1 = field.
ELSEIF rec_cnt EQ 2.
field2 = field.
ELSEIF rec_cnt EQ 3.
field3 = field.
ELSEIF rec_cnt EQ 4.
field4 = field.
ENDIF.
ENDIF.
ENDIF.
ENDIF.
CONCATENATE field str+cnt(1) INTO field.
cnt = cnt + 1.
IF cnt GE len.
EXIT.
ENDIF.
ENDDO.
WRITE: field1, field2, field3, field4.
ENDFORM.
Regards,
Wenceslaus. -
How to convert xslt file into string
i'm writting a java program to use xslt to transform the xml file. i'm encountering the problem when i try to convert the xslt file into string. i've defined my utility class called 'XmlUtil' to carry out the operation of transform xml file through xslt. but in my main java program i need to convert both xml and xslt file into a string in order to input them in my function argument. my function argument is as follows:
String htmlString = XmlUtil.applyXsltString(xmlContent, xsltString);
i've already converted xmlcontent into string by using:
xmlContent = xmlContentBuffer.toString();
but i don't know how to convert 'xsltString' now ? i've searched the google for an hour but i cannot find the solution. anyone can help ?
detail of my souce code is as follow:
import java.io.*;
import java.net.*;
import java.lang.*;
import java.io.StringReader;
import java.lang.reflect.Array;
import java.util.Properties;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.sax.SAXResult;
import javax.xml.transform.sax.SAXSource;
import javax.xml.transform.sax.SAXTransformerFactory;
import javax.xml.transform.sax.TransformerHandler;
import javax.xml.transform.stream.StreamSource;
import org.apache.xml.serializer.OutputPropertiesFactory;
import org.apache.xml.serializer.Serializer;
import org.apache.xml.serializer.SerializerFactory;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.XMLReaderFactory;
import XmlUtil;
public class FileDownload {
public static void download(String address, String localFileName){
OutputStream out = null;
URLConnection conn = null;
InputStream in = null;
StringBuffer xmlContentBuffer = new StringBuffer();
String temp = new String();
String xmlContent;
try {
URL url = new URL(address);
out = new BufferedOutputStream(
new FileOutputStream(localFileName));
conn = url.openConnection();
in = conn.getInputStream();
byte[] buffer = new byte[1024];
int numRead;
long numWritten = 0;
System.out.println (in.toString ());
while ((numRead = in.read(buffer)) != -1) {
out.write(buffer, 0, numRead);
numWritten += numRead;
temp = new String(buffer);
xmlContentBuffer.append(temp);
System.out.println(localFileName + "\t" + numWritten);
xmlContent = xmlContentBuffer.toString();
String htmlString = XmlUtil.applyXsltString(xmlContent, xsltString);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
if (out != null) {
out.close();
} catch (IOException ioe) {
public static void download(String address) {
int lastSlashIndex = address.lastIndexOf('/');
if (lastSlashIndex >= 0 &&
lastSlashIndex < address.length() - 1) {
download(address, address.substring(lastSlashIndex + 1));
} else {
System.err.println("Could not figure out local file name for " + address);
public static void main(String[] args) {
for (int i = 0; i < args.length; i++) {
download(args);
}I don't understand why you need load the XML and XLS files into a String. A Transformer can be constructed from a Source and there is a StreamSouce which can be constructed from an InputStream. The transform() method can take a Source as input and can produce a Result. There is no need to go near a String representation of either the input.
-
Unable to read payload from the message object
Hi
I have a scenario where i am send request to http receiver and getting the response. When I am testing through WFETCH it is working fine. But when i am testing through XI I am getting the follwoing error
Unable to read payload from the message object
I have tested the XI payload in mapping. I have done all kinds of testing but it is still giving the same error.
One more strange thing is
I have done one BPM scenario where Data is coming from Source to BPM( which is asyn) and then from it will go from BPM to Target (which is sync) But when I am checking the SXMB_MONI... it showing the messages like this
Source to BPM
Target to BPM
Target to BPM.
But i think it should show message like
Source to BPM
BPM to Target
Target to BPM
why i am getting the floHi
Check with the connection string & Authorization of user you have used.
MSSQL string is
jdbc:microsoft:sqlserver://dbhost:1433;databaseName=example;SelectMethod=Cursor
While doing JDBC its good to refer Note 831162 lot of problems can be resolved.
Thanks
Gaurav -
Spilt a big file into 3 files??
Hello,
Is there any software I can use to spilt a very big file into
3 or 4 small files please?
Thanks.Use one of these applications to split the file.
(14020) -
How to split a big file into several small files using XI
Hi,
Is there any way to split a huge file into small files using XI.
Thanks
MukeshThere is a how to guide for using file adapters, an "sdn search" will get you the document.
Based on that , read a file into XI, Use strings to store the file content and do the split
here is some code to get you started
===========
Pseudocode:
===========
// This can be passed in - however many output files to which the source is split
numOutputFiles = 5;
// Create the same number of filehandles as numOutputFiles specifies
open file1, file2, file3, file4, file5;
// Create an Array to hold the references to those filehandles
Array[5] fileHandles = {file1, file2, file3, file5, file5};
// Initialize a loop counter
int loopCounter = 0;
// a temporary holder to "point" the output stream to the correct output file
File currentOutputFile = null;
// loop through your input file
while (sourceFile.nextLine != null)
loopCounter++;
if (loopCounter == (numOutputFiles +1) // you've reached 5, loop back
loopCounter = 1;
currentOutputFile = fileHandles[loopCounter]; // gets the output file at that index of the array
currentOutputFile.write(sourceFile.nextLine);
regards
krishna
Maybe you are looking for
-
Spry Horizontal Menu Submenus Won't Appear - Help
OK, I have a Spry Horizontal Menu Bar done through DW CS5. The root menu items work just fine. And the arrow graphics indicating there are submenus are shown, however when you hover over the main menu item the submenus do not appear in browser previe
-
How to define Export parameter as Import parameter in next test script.
Hi Experts, We have defined Purchase order no. as export parameter in ME21N transaction test script. Test configuration for ME21N transaction executed succesfully and the Purchase order No. appeared in the last message is the export parameter for nex
-
Blurry or Fuzzy Text & Headings???
I use Acrobat Professional 7.1 and it has worked and printed properly until the last couple days. Something has recently changed and now my printing is blurry... When I receive a pdf and go to print it out, the preview looks normal. When I print, if
-
Including Trusted sites for flash
In flex help documentation it is mentioned that if swf file is loaded from a website which is listed trusted in browser, then it acts as if file in local trusted sandbox. But it does not work. Has anyone tried it? Here is the help quote: "Trusted sit
-
All, I have ALV grid using CONTAINER , i need to make it FULL SCREEN mode as per the user screen display resolution. Any Info ? PS: Please donot suggest to drag the screen to maximum. that will not work with different screen resolution. a®