Searching through very large vectors

Question

I am working on a way to process two flat tab delimited files into a tree, assign a x and y coordinate to each node in the tree and output all the nodes (with their coordinates) to a new flat file.
I currently have a program that works pretty well. It roughly uses the following flow.
- Read both files into memory (by opening the file reading each line and load the appropriate data from each line into a Vector, making sure no duplicates are entered by comparing the currentline to the last line.
- Using the first vector (which contains the strating nodes) search through the second vector (which contains parent child relationships between 2 nodes) to construct the tree. For this tree I use a XML DOM Document. In this logic I use a for loop to find all the children for the given node. I store the index of each found reference and when all children are found I loop through all the indexes and delete those records from the parent-child vector.
- After the tree is created I walk through the tree and assign each node a x and y attribute.
- When this is done I create a NodeList and use are for-loop to write each node (with x and y) to a StringBuffer which is then written to a file. In this process for each new node that is written I check (in the StringBuffer) if the node (name) is present. If not I write the new Node.
- For debugging purposes I write all the references from the second Vector to a file and output the XML DOM tree to a XML file.
This program works wel. It handles files with 10000 start nodes and 20000 parent-child references (30000 nodes in total) in under 2 minutes (using 1:20 for the generation of the output file).
However when the volume of these file increase it starts to struggle.
As the ultimate test I ran it with a file that contains 250000 start nodes and 500000 references. For it to run I need to use the -Xmx256m parameter to allocate extra memory. But I ran it for 2 hours and killed it because I didn't want to wait longer.
What I would like to know is how I can approach this better. Right now I'm loading the data from the files into memory entirely. Maybe this isn't the best approach.
Also I'm looping through a Vector with 500000 elements, how can this be done more efficiently? However the reference vector isn't sorted in any way.

Answer 1

Hi,
That's no problem.. Here's some sample code:
package tests;
import java.util.List;
import java.util.Map;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Iterator;
class Example {
    private List roots;
    private Map elements;
    public Example() {
        roots = new LinkedList();
        elements = new HashMap();
    public void initRoots(String[] rows) {
        for (int i=0; i<rows.length; i++) {
            String[] parts = rows.split(" ");
String name = parts[0];
roots.add(name);
elements.put(name, new Node(name));
public void addChilds(String[] rows) {
for (int i=0; i<rows.length; i++) {
String[] parts = rows[i].split(" ");
String parentId = parts[1];
String name = parts[2];
addNode(parentId, name);
private void addNode(String parentId, String name) {
Node current = (Node)elements.get(name);
if (current == null) {
current = new Node(name);
elements.put(name, current);
Node parent = (Node)elements.get(parentId);
if (parent == null) {
//Parent is missing, is that a problem?. Create it now.
parent = new Node(parentId);
elements.put(parentId, parent);
return;
parent.addChild(current);
public void printTree() {
for (Iterator it = roots.iterator(); it.hasNext(); ) {
String id = (String)it.next();
printChildren(id, 1);
private void printChildren(String id, int depth) {
Node node = (Node)elements.get(id);
System.out.println(node);
private static final class Node {
private String name;
private List children;
private Node(String name) {
this.name = name;
children = new LinkedList();
public void addChild(Node node) {
children.add(node);
public String toString() {
return name + " " + children;
public static void main(String[] args) throws Exception {
Example test = new Example();
test.initRoots(new String[] {
"SU_1 1 1 1 0 0 0 0",
"SU_2 1 1 1 0 0 0 0",
"SU_3 1 1 1 0 0 0 0"
test.addChilds(new String[] {
"COM_1 SU_1 PR_1 0 0 0 0 0",
"COM_1 PR_1 ST_1 0 0 0 0 0",
"COM_2 SU_2 PR_2 0 0 0 0 0",
"COM_2 PR_2 ST_2 0 0 0 0 0",
"COM_3 SU_3 PR_3 0 0 0 0 0",
"COM_3 PR_3 ST_3 0 0 0 0 0"
test.printTree();
The execution prints:
SU_1 [PR_1 [ST_1 []]]
SU_2 [PR_2 [ST_2 []]]
SU_3 [PR_3 [ST_3 []]]
/Kaj