How to make program eat less memory/enhance performance ?

Question

Dear experts,
I have a program that creates 3 tables containing series of line vectors. These are created from data contained in an ascii file. The program performs some calculations on the data. I have everything working perfectly i.e. it creates the tables, does the calculations and gives me the final results which I need (but this only happens when I use a data file that is 88Kb in size).
However, the frustrating problem I am experiencing currently is that when I use a heavier file e.g size 4.22mb, after a short delay, Windows starts complaining about memory errors and Windows task manager's Mem usage goes through the roof!
Eventually, I get an Exception in thread ?main? java.lang.OutOfMemoryError <<no stack trace available>> error from the JVM.
I have tried "java -Xrunhprof" command and it suggests that the real memory hogging line of code is:
name = ((String) v.elementAt(nj))+"\t"+((String) reference_vec.elementAt(nj));As the program is dynamically allocating and creating a full nxn matrix of the input data lines at run-time that means it is creating zillions of strings at that line? Concatenation of Strings is not a good idea either as this blows performance too. I am thinking of re-writing the program in C. I have run out of ideas. Please can you suggest how I can make some tweaks to the java code so that the program runs more efficiently? Any comments welcome. Program is approx 500 LOC, can post on request.
d.

Answer 1

import java.io.*;
import java.util.*;
          Mobile ID Classification Program - 2003
          Currently known memory bugs -
          15/10/03      -          JVM reports OutOfMemoryError when gsm_data.txt used as input
public class DataAnalysis {
/* Data structures used to store values **/
               private Hashtable Activity = null;
               private Vector table1 = null;
               private Vector table2 = null;
               private Vector table3 = null;
               private BufferedReader fin;
               private StringTokenizer st;
     Mobile activity description strings       
     private final String A1 = "IMSI attach";
     private final String A2 = "IMSI detach";
     private final String A3 = "Periodic location update";
     private final String A4 = "Normal location update+authentication";
     private final String A5 = "Service request mobile call";
     private final String A6 = "Service request SMS";
     private final String A7 = "Service request supplementary service activation";
     private final String A8 = "paging response";
     //           private final String A8a = "Paging response+authentication";
     private final String A9 = "Service request energy";
/*     main function where the program begins execution **/
     public static void main(String args[]) {
               if(args.length == 0) {
                         System.out.println("File name not specified. Usage : java -Xmx500mb DataAnalysis <file name> ");
          else if(args.length > 1) {
                         System.out.println("Too many parameters. Usage : java -Xmx500mb DataAnalysis <file name> ");
          else {
                              new DataAnalysis(args[0]);
/*          Constructor. Takes input file name string as parameter      **/
     public DataAnalysis(String data_file_name) {
               Activity = new Hashtable();
                                        table1 = new Vector();
                                        table2 = new Vector();
                                        table3 = new Vector();
     //                Parse the input file. Read in lines.
          try{
               fin= new BufferedReader(new FileReader(data_file_name));
               String line;
               while((line = fin.readLine()) != null) {
                    if (line.trim().length() > 0) {
                         tokenize(line);
          catch(Exception e) {
                              System.out.println(e.getMessage());
          finally {
               try {
                              fin.close();                    //       Close the file input stream then 
               catch(Exception ee) {
               //           Create the first two tables
                                                       createTable1();
                                                       createTable2();
          //                     Free the memory occupied by table 1
          //                    freeTable(table1);
                    table1 = null;
                                                       createTable3();
                         //                Free the memory occupied by table 2          
                         //                freeTable(table2);
               table2 = null;
          //                Select all the mobile ids in gsm_data.txt which have a city block distance value less than 0.25 threshold (upper limit)
          System.out.println("--------------------------Table 3 : Relationship between mobile ids using Manhattan City Block distance -------------------------- \n" );
          for(int ni =0;ni<table3.size();ni++) {
               Vector inner_vec = (Vector) table3.elementAt(ni);
               for(int nj = 0;nj<(inner_vec.size()-1);nj= nj+2) {
                         float val = ((Float) inner_vec.elementAt(nj)).floatValue();
                         if(val <= 0.25 && val > 0.0)  {                              
                                        System.out.println((String) inner_vec.elementAt(nj+1));
/*      Returns the index from table 1  **/
     public int getIndexFromTable1(String id) {
          int index = -1;
          for(int ni = 0;ni<table1.size();ni++) {
               Vector v = (Vector) table1.elementAt(ni);
               String vect_id = (String) v.elementAt(0);
               if(vect_id.equalsIgnoreCase(id)) {
                         index = ni;
                         return index;
                    Create table with actual frequency of mobile activity for each mobile id.   
                    Number of times an activity occurred in the input file.          
     public void createTable1() {
                    Enumeration e = Activity.keys();
                    int sum;
          while(e.hasMoreElements()) {
                    Vector table1_row = null;
                    String s = (String) e.nextElement();
                    Integer num = (Integer) Activity.get(s);
                    StringTokenizer st1 = new StringTokenizer(s, "|");
                    String mobile_id = (String) st1.nextToken();
                    String mobile_act = (String) st1.nextToken();
                    int row_index = getIndexFromTable1(mobile_id);
               if(row_index > 0) {
                              table1_row = (Vector) table1.elementAt(row_index);
                              sum = ((Integer) table1_row.lastElement()).intValue();
               else {
                                   table1_row = new Vector();
                                   sum = 0;                    
               if(table1_row.contains(mobile_id)) {
                    if(mobile_act.equalsIgnoreCase(A1)) {                              //                IMSI attach activity
                              table1_row.setElementAt(num, 1);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A2)) {                    //                IMSI detach activity
                              table1_row.setElementAt(num, 2);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A3)) {                    //                Periodic location update activity
                              table1_row.setElementAt(num, 3);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A4)) {                    //            Normal location update+authentication activity
                              table1_row.setElementAt(num, 4);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A5)) {                    //            Service request mobile call activity
                              table1_row.setElementAt(num, 5);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A6)) {                    //                Service request SMS activity
                              table1_row.setElementAt(num, 6);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A7)) {                    //                Service request supplementary service activation activity
                              table1_row.setElementAt(num, 7);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A8)) {               //                     Paging response activity                         
                              table1_row.setElementAt(num, 8);
                              sum += num.intValue();
                    else if(mobile_act.equalsIgnoreCase(A9)) {               //                    Service request energy activity          
                              table1_row.setElementAt(num, 9);
                              sum += num.intValue();
                              int lastIndex = table1_row.lastIndexOf(table1_row.lastElement());               //       This is the last index which this table
                              table1_row.setElementAt(new Integer(sum), lastIndex);
               else {
                    table1_row.addElement(mobile_id);
                    if(mobile_act.equalsIgnoreCase(A1)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A2)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A3)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A4)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A5)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A6)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A7)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                              table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A8)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                         table1_row.addElement(new Integer("0"));
                    if(mobile_act.equalsIgnoreCase(A9)) {
                         table1_row.addElement(num);
                         sum += num.intValue();
                    else {
                                   table1_row.addElement(new Integer("0"));
                                   table1_row.addElement(new Integer(sum));
               if(row_index > 0){
                    //          Replace the vector in table1 with updated vector
                                        table1.remove(row_index);
                                        table1.insertElementAt(table1_row, row_index);
               else {
                                   //                 Add the row vector to the table vector as a row.
                                                       table1.addElement(table1_row);
               tableToString(table1, "--------------------------Table 1: Number of times activity occurred in input txt file -------------------------- " );
                    Create table 2 by normalising the data values from table 1 
     public void createTable2() {
          for(int ni = 0;ni<table1.size();ni++) {
                    int sum;
                    String name = "";
                                        Vector table2_row = new Vector();
                                        Vector table1_row = (Vector) table1.elementAt(ni);
               //            For each record of table 1
               for(int nj = 0;nj<table1_row.size();nj++) {
                    sum = ((Integer) table1_row.lastElement()).intValue();
                    //               Name of the activity
                    if(nj == 0) {
                                   name = (String) table1_row.elementAt(nj);
                                   table2_row.addElement(table1_row.elementAt(nj));
                    else if (nj == (table1_row.size() - 1)) {
                              //                Don't do any thing, we dont want the sum of activities in table 2 !
                    else {
                         Integer num = (Integer) table1_row.elementAt(nj);
                         int num_val = num.intValue();
                         if(num_val == 0) {
                                        table2_row.addElement(new Float("0.0"));
                         else {
                              if(sum > 0.0) {
                                        table2_row.addElement(new Float((float)num_val/sum));
                              else {
                                        table2_row.addElement(new Float("0.0"));
                    table2.addElement(table2_row);
                                        tableToString(table2, "--------------------------Table 2: Table 1's Normalised Values --------------------------");
          Create table 3 by comparing the relationship between vectors of table 2 using
          the Manhattan City Block distance.
     public void createTable3() {
               Vector reference_vec = null;
          for(int ni = 0;ni<table2.size();ni++) {
                              reference_vec = (Vector)table2.elementAt(ni);
                              createTable3Record(reference_vec);
/*       public void createTable3Record(Vector reference_vec) {
          //       Vector table3_record = new Vector();
          Table3Record table3_record = new Table3Record();
          for(int ni = 0;ni<table2.size();ni++) {
               float sum = 0;
               //           String name = "";
               Vector v = (Vector) table2.elementAt(ni);        //           each record of table2
               for(int nj = 0;nj<v.size();nj++) {                    //           for each element of a record of table 2. Iterate through table 2.
                    if (nj == 0) {
                         //           name = ((String) v.elementAt(nj))+"\t"+((String) reference_vec.elementAt(nj));
                                        table3_record.name_prefix = (String) v.elementAt(nj);
                                        table3_record.name_suffix = (String) reference_vec.elementAt(nj);
                                             //       table3_record.value = sum;                         
                    else {
                                             float f1 = ((Float) v.elementAt(nj)).floatValue();
                                             float f2 = ((Float) reference_vec.elementAt(nj)).floatValue();
                              sum += Math.abs(f1-f2);
                              table3_record.value = sum;
                                   //               table3_record.addElement(new Float(table3_record.value));                    //                Value
                                   //             table3_record.addElement(table3_record.name);                                             //                Vector Names for that value
                                                  table3.addElement(table3_record);
                                   //                table3_record = null;                         
               OLD CODE BUT AT LEAST IT WORKS !!  //    Creates a record for the third table            //  OLD CODE BUT IT WORKS
     public void createTable3Record(Vector reference_vec) {
                    Vector table3_record = new Vector();
          for(int ni = 0;ni<table2.size();ni++) {
                    float sum = 0;
                    String name = "";
                    Vector v = (Vector) table2.elementAt(ni);             //           each record of table2
               for(int nj = 0;nj<v.size();nj++) {                    //           for each element of a record of table 2. Iterate through table 2.
                         if (nj == 0) {
                          name = ((String) v.elementAt(nj))+"\t"+((String) reference_vec.elementAt(nj));               /**  MEMORY BUG  ?   */
                              //name = new StringBuffer().append((String) v.elementAt(nj)).append("\t").append((String) reference_vec.elementAt(nj)).toString();               
                    else {
                                        float f1 = ((Float) v.elementAt(nj)).floatValue();
                                        float f2 = ((Float) reference_vec.elementAt(nj)).floatValue();
                                        sum += Math.abs(f1-f2);                              //            Calculate the sum
                                        table3_record.addElement(new Float(sum));                    //           Value
                                        table3_record.addElement(name);                                             //           Vector Names for that value
                                        table3.addElement(table3_record);
                                        table3_record = null;
               Method which breaks the contents of the file which is read into memory for processing 
     public void tokenize(String s) {
                              st = new StringTokenizer(s, "\t");
                              String id = "";
                              //       String time = "";
                              String activity = "";
          if(st.hasMoreTokens()) {
                                        st.nextToken();                                                            //                      Date e.g. 13052003
                                        //           time = st.nextToken();                                                            //            Time          e.g. 11:30:18
                                        st.nextToken();
                                        st.nextToken();                                                            //                      Frame Number e.g. 166827                              
                                        id = st.nextToken().trim();                                   //                          ID e.g. f47c12cffb or 2943012109009630
                                        st.nextToken();                                                            //                          Classmark  e.g 33
                                        st.nextToken();                                                            //                          Float e.g. 2.2  (the distance from base station)
                                        activity = st.nextToken().trim();                    //                      Activity e.g IMSI attach
               String key = id+"|"+activity;
                                        key = key.intern();               
               if (Activity.containsKey(key)) {
                                   Integer num = (Integer) Activity.get(key);
                                   int num_val = num.intValue();
                                   num_val += 1;
                                   num = new Integer(num_val);
                                   Activity.put(key, num);
               else {
                         Activity.put(key, new Integer(1));               //  Insert key into our Hashtable
               Frees the memory used by a table 
     public void freeTable(Vector table) {
                    for (int ni = 0;ni<table.size();ni++) {
                                        Vector inner = (Vector) table.elementAt(ni);
                                        inner = null;
                                        table = null;
               Convert a vector to a string suitable for displaying on console output               
     public void tableToString(Vector table, String table_name){
               System.out.println(table_name + "\n");
                              for (Iterator it = table.iterator(); it.hasNext();) {                                                              
                                             System.out.println(it.next());
               Function which sepearates mobile id usage beween day/night usage.     
     public void calculateTimeInterval(String t1, String t2){
                         //           t' = t1 - t2          
                         //           Look at all the ids
          Date d1,d2;
                         d1 = new Date(t1);
                         d2 = new Date(t2);
                         // TBD
}