OutOfMemory-help on memory saving tips

Question

Hello,
I am fairly new with Java.
My software ran out of memory when I tried to process much more input than it had ever done before.
I realized that one of the problems was that it woudl have been a good idea to specify the size of the big ArrayLists I use to prevent them from allocating more memory than necessary. BTW, I already use the -Xms -Xmx options when running java.
Unfortunately, I can't see any other way of saving memory. I paste below the relevant sections of code where I think there are improvements to be made and my comments/questions. I would be really grateful if you could help me in this.
Piece of code number 1:
HashMap srcToNewCand = new HashMap(35000);
//I would need only 25000 key-value pairs, but I do not want the HashMap to start reloading for space when it is almost full. So, I declared it to be initially of 35000 key-value pairs. Was that a bad idea ?
debug("Loading dictionary-based candidates...");
if (extendListCandidates) {
RerankInstance [] newCandExamples = new RerankInstance[examples.length+10000];
newCandExamples = TestListPreparation_G.readExamples(new File (newCandPath));
System.out.println ("...Check this out: how many examples..." + newCandExamples.length);
for (int j = 0; j < newCandExamples.length; j++) {
String src = newCandExamples[j].getSourcePhrase().getPhrase();
for (Iterator it = newCandExamples[j].getCandidates(); it.hasNext();) {
CandidatePhrase c = (CandidatePhrase)it.next();
// System.out.println("THERE " + c.getPhrase());
srcToNewCand.put(src, newCandExamples[j]);
=======================================
Piece of code number 2:
public static RerankInstance [] readExamples(File in)
throws IOException
int initialSize = 25500; //There will actually be 25000 objects in the ArrayList ret
ArrayList ret = new ArrayList(initialSize);
BufferedReader r = new BufferedReader(new InputStreamReader(new FileInputStream ...
String line, spanish=null, gold = null;
ArrayList candidates = new ArrayList(); //There can be at most 205 elements, and in
//cases less than 205. So, I am not sure that is is wise to
//establish from the beginning the initial size of ArrayList candidates to 205.
ArrayList LM_likelihoods = new ArrayList(); //Same for these ArrayLists
ArrayList TM_likelihoods = new ArrayList();
ArrayList LM_TM_likelihoods = new ArrayList();
Double lm = null;
Double tm = null;
Double lm_tm = null;
boolean readingCandidates = false;
RerankInstance current = null;
int cnt = 0;
while ( (line = r.readLine()) != null)
if (line.equals("<REC>"))
if (line.equals("</REC>"))
if (spanish != null && candidates.size() > 0)
System.out.println("Spanish=" + spanish);
if (++cnt % 10 == 0)
System.out.print(cnt + " ");
String [] cands = (String [])
candidates.toArray(new String[0]);
RerankInstance foo = new RerankInstance(spanish, cands);
//How important would it be for memory saving not to create LM_likes below ?
if (LM_likelihoods.size() > 0 ){
Double [] LM_likes = (Double[])LM_likelihoods.toArray(new Double[0]);
foo.setLikelihoods_single(LM_likes);
if (TM_likelihoods.size() > 0 ){
Double [] TM_likes = (Double[])TM_likelihoods.toArray(new Double[0]);
foo.setLikelihoods_single(TM_likes);
if (LM_TM_likelihoods.size() > 0 ) {
Double [] LM_TM_likes = (Double[])LM_TM_likelihoods.toArray(new Double[0]);
foo.setLikelihoods_single(LM_TM_likes);
foo.read_setLikelihoods(foo,LM_likelihoods,TM_likelihoods, LM_TM_likelihoods);
ret.add(foo);
foo.setGold(gold);
spanish = null;
gold = null;
candidates.clear();
LM_likelihoods.clear();
TM_likelihoods.clear();
LM_TM_likelihoods.clear();
r.close();
System.out.println("RET LENGTH " + ret.size());
System.gc();
return (RerankInstance [])ret.toArray(new RerankInstance[0]);

Answer 1

I guess, this is the problem, but I need to process the files and put every chunk into an object RerankInstance. And each RerankInstance into the array for further processing ...
Is it the array size or reading the file ? Or both ? This is the first time I deal with these huge files.
What are the techniques for dealing with them ?
The file is a set of sequences like this one between <REC> and </REC> and I load each of them into RerankInstance objects, which I report below, just in case :
<REC>
Spanish=dimensi�n patrimonial
Gold=part
<CANDIDATES>
heritage -21.487333459850646 -18.621516230769643 -40.10884999269363
heritage ' -24.333740847686165 -18.829122778712982 -43.1628646120767
dimension heritage -33.36906589697976 -9.977052287654427 -43.34612137267963
scale heritage -31.950884548302326 -12.290624504161759 -44.241509982724374
i heritage -26.1880298363302 -19.114430105787065 -45.30246266721862
aspect heritage -32.913069542285406 -13.020032446306868 -45.933101380644814
size heritage -34.23091676545765 -11.87414357059991 -46.1050636078338
dimension heritage ' -36.215471186538686 -10.184660190733727 -46.4001337286498
dimensions heritage -34.64893953584428 -11.985933603236262 -46.634874187232285
dimension to heritage -36.636846373518104 -10.292145692327923 -46.92899305690606
extent heritage -32.17625397505681 -14.979772384720786 -47.156025426894395
element heritage -33.964103113598426 -14.16680002962667 -48.13090343035614
level heritage -31.662813503493684 -16.712797571218466 -48.37560967492999
i dimension heritage -38.06976360615384 -10.469966840492953 -48.53973138075892
side heritage -34.12488687204262 -15.187121253699013 -49.31200402851826
mafia heritage -33.307202042321556 -16.785347040598296 -50.09254736220585
pecuniary heritage -36.844249654645914 -13.510639565091813 -50.35488773970646
aspects heritage -36.14221235514915 -15.78933881593482 -51.93154990091026
</CANDIDATES>
</REC>
public class RerankInstance
private CandidatePhrase srcPhrase;
private LinkedHashMap candidates = new LinkedHashMap();
private String gold = null;
public RerankInstance(String sourcePhrase,
String [] candidatePhrases)
boolean flag = false;
ISIPhrase source = new ISIPhrase(sourcePhrase, flag);
//this.srcPhrase = makeCandidatePhrase(source);
this.srcPhrase = new CandidatePhrase(source);
candidates = new LinkedHashMap();
for (int i = 0; i < candidatePhrases.length; i++)
if (candidatePhrases[i] == null ||
candidatePhrases.trim().length() == 0)
continue;
ISIPhrase temp = new ISIPhrase(candidatePhrases[i], false);
if(candidates.containsKey(temp.getPhrase()))
temp.setIgnore(true);
flag = true;
CandidatePhrase cp = new CandidatePhrase(temp);
//CandidatePhrase cp = makeCandidatePhrase(temp);
if (cp == null)
continue;
candidates.put(temp, cp);
// methods............
}//end of class