Need help on efficient searches using hashes on large #s of data structures

Question

I have a massive amount of data and I need a way to access it using generated indexes instead of traditional searching such as binary chop as I require a fast response time.
I have 100,000 Array data structures each containing roughly 10 to 60 elements. The elements of the Arrays do not store any raw data they just hold references to objects. I have about 10,000 unique objects which are referenced to by the arrays.
What I need to do is supply a number of objects as the query and the system should return all the arrays which contain two or more different objects from the query.
So for example>
Given query objects> “DER” , “ERE” , “YPS”
and arrays> [DER, SFR, PPR]
     [PER, ERE, SWE, YPS]
     [ERE, PPD, DER, YPS, SWE]     
     [PRD, LDF, WSA, MMD]
It should return arrays 2 and 3.
I had a look at hashMaps but I’m not entirely sure if that’s what I need.
Would I need to hash the each Array and then hash the query objects and look for overlaps in the two hashes or something similar?
Thanks In Advance

Answer 1

For a single query, it's more efficient to do less work, so don't create intermediate objects, don't create a index of objects to sets of possibles, and stop counting matches when you find enough:.
                final HashSet<String> querySet = new HashSet<String>(Arrays.asList(query));
                final Set<String> results = new LinkedHashSet<String>();
                // find matches using scan of all arrays
                for (String[] array : data) {
                    int matchCount = 0;
                    for (String elt : array) {
                        if (querySet.contains(elt)) {
                            ++matchCount;
                            if (matchCount >= 2) {
                                results.add(Arrays.asList(array).toString());
                                break;
                }Using random data, 100,000 arrays of selections from a list of 10,000 trigrams, scanning all the arrays takes about as long as the set code, but doesn't require generation of the map, which can take up a lot of memory if you pre-create the sets (I didn't complete a run doing that, although it should be faster to query, it used too much memory and my computer started swapping so I gave up after a few minutes).
Running queries of size 2 to 25, times in ms:
linear scan of all arrays in data set:
elapsed: 2196
creating index of trigram to arrays containing trigram:
elapsed: 4635
query using set union and index lookup:
elapsed: 2195
query using direct scan and index lookup:
elapsed: 164
So if I was doing one query per dataset, just use a simple scan as above. If you're doing lots on the same dataset, then create a map<String, List<String[]>> to index the arrays, and the lookup will save quite a bit of time.
Creating the index:.
            dataSets = new HashMap<String, List<String[]>>();
            for (String trigram : trigrams)
                dataSets.put(trigram, new ArrayList<String[]>());
            for (String[] array : data)
                for (String trigram : array)
                    dataSets.get(trigram).add(array);Finding matches using scan of arrays with at least one elt in query:.
                final HashSet<String> querySet = new HashSet<String>(Arrays.asList(query));
                final Set<String> results = new LinkedHashSet<String>();
                for (String trigram : query) {
                    for (String[] array : dataSets.get(trigram)) {
                        for (String elt : array) {
                            if (elt != trigram && querySet.contains(elt)) {
                                results.add(Arrays.asList(array).toString());
                                break;
                }