ArrayList indices (Phrase counter follow-up)

Question

Hi everybody--
I finally figured out how to get code onto this computer I'm working with, so here's what I'm trying to do for anyone that doesn't remember. I'm using TuringPest's suggestion to go through my word list once, and then delete any index that fails on an expansion.
This is my original post:
I'm writing a phrase counter, and so far it's working, but I'm trying to optimize it because running over a for loop is really slowing me down.
Here's the problem:
I want to get phrases that repeat more than once, from length 3 words to 10 words. Each time I change the length of the phrase (say, from 10 to 9 to 8, etc down to 3), I have to run over the entire word list all over again, which eventually amounts to 8 total passes over an ArrayList that's almost 45000 elements long right now, and will get longer.
An added complication in this is that there are certain identifiers in the word list that mark the beginnings of chapters (#) and sentences ($); any time they are hit, the iterator moves to the next word.
So, basically, something like...
example example example example example $ # example example example $
...with the repeating loop I have now would give:
0 phrases of length 10, 9, 8, 7, or 6
1 of length 5
2 of length 4
4 of length 3
and this is the reply I'm trying to emulate:
Find the occurences of the smaller phrases first.
Index their locations.
Do all subsequent longer phrase matches only from the saved locations.
I wrote a self-contained test program, and the problem with it seems to be that the correct indices aren't being saved. The first iteration where we go through every possible index for phrases 3 words long works fine, but every subsequent one doesn't and I'm not sure why. Here's the code:
EDIT: Sorry about the bump but I just realized I might have left something out for clarity--the newIndexes list is being created each time to hold the places where phrases can still be built, and then replacing the old indexes list (at least, that's what it's supposed to do).
import java.util.ArrayList;
import java.util.TreeMap;
public class Test {
     public static void main(String[] args) {
          ArrayList<String> words = new ArrayList<String>();
                // Two different arrays to test with
          //String[] raw = {"#", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "$", "#", "eleven", "twelve", "thirteen", "fourteen", "$"};
          String[] raw = {"#", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "$"};
          for( int i = 0; i < raw.length; i++ ) {
               words.add( raw[i] );
          TreeMap<String, ArrayList<Integer>> phraseList = new TreeMap<String, ArrayList<Integer>>();
          ArrayList<Integer> indexes = new ArrayList<Integer>();
          int lengthOfPhrase = 3;
          for( int i = 0; i < words.size(); i++ ) {
               indexes.add( i );
          do {
               TreeMap<String, ArrayList<Integer>> phraseRaw = new TreeMap<String, ArrayList<Integer>>();
               ArrayList<Integer> newIndexes = new ArrayList<Integer>();
               int chapter = 0;
               for( int i = 0; i < indexes.size(); i++ ) {
                    String ready = words.get( indexes.get( i ) );
                    boolean okay = true;
                    if( ready.equalsIgnoreCase( "#" ) ) {
                         okay = false;
                    } else if( !ready.equalsIgnoreCase( "$" ) ) {
                         okay = true;
                         for( int j = 0; j < lengthOfPhrase; j++ ) {
                              if( j != 0 ) {
                                   ready = ready.concat( " " );
                                   String next = words.get( i + j );
                                   if( next.equalsIgnoreCase( "#" ) || next.equalsIgnoreCase( "$" ) ) {
                                        okay = false;
                                        break;
                                   } else {
                                        ready = ready.concat( next );
                         if( okay ) {
                              newIndexes.add( indexes.get( i ) );
                              if( phraseList.containsKey( ready ) ) {
                                   ArrayList<Integer> values = phraseList.get( ready );
                                   values.add( chapter );
                                   phraseList.put( ready, values );
                              } else {
                                   if( phraseRaw.containsKey( ready ) ) {
                                        ArrayList<Integer> values = phraseRaw.get( ready );
                                        values.add( chapter );
                                        phraseList.put( ready, values );
                                   } else {
                                        ArrayList<Integer> newValues = new ArrayList<Integer>();
                                        newValues.add( chapter );
                                        phraseRaw.put( ready, newValues );
               lengthOfPhrase++;
               indexes.clear();
               indexes = newIndexes;
          } while( lengthOfPhrase <= 10 );
}Does anyone see where I'm going wrong?
Thanks,
Jezzica85
Message was edited by:
jezzica85

Answer 1

OK, I guess I'll try again to explain what's going on and try to be more clear.
Inside the map, there is a list of phrases, ranging from 3 to 20 words in length. For those phrases that appear in the document more than once (that is, the length of their arrayList value is 2 or more), I need to know if they are contained within any other phrase keys in the map.
So, if part of the map was like this:
This is an example=[2,4,6]
This is an=[2,4,6]
is another example of this=[1,3,5]
is another example=[1,3,5]
The revised map after the substrings were taken out would be:
This is an example=[2,4,6]
is another example of this=[1,3,5]
The second entry of the original map would be taken out because it was contained by and occurred in the same positions as the first, and the fourth entry of the original map would be taken out because it was contained by and occurred in the same positions as the third.
Is this any clearer?
Thanks for looking and trying to help,
Jezzica85