NOT operator: What kind of index to create to prevent unindexed searches ?

Question

I see the following unindexed searches in my ODSEE (dsconf : 11.1.1.3.0 B2010.0630.2145) directory access log:
[30/Sep/2011:11:42:04 +0200] conn=133 op=1 msgId=2 - SRCH base="ou=people,ou=.... " scope=1 filter="(!(employeeType=P))" attrs="mail sn givenName"
[30/Sep/2011:11:42:28 +0200] conn=133 op=1 msgId=2 - RESULT err=0 tag=101 nentries=94312 etime=24 notes=U
I've created an equality index on the employeeType attribute with a threshold of 130 000. I know that I have 4 possible values for the
employeetype attribute, where at most 90000 entries have the same value. So, I don't understand why it's considered as an unindexed search.
What index(es) and index parameter(s) should I create to have that kind of search match an index ?

Answer 1

A while back Ludovic (I think) posted that there is no index for NOT. If you think about it, the usual index operation for equality, for example, has a key and value. The employeeType equality index key for "P" would have as its value a list of entryIDs that have the appropriate employeetype. So where would we get the list of entryIDs that don't have the value? Every key would need a corresponding negation key, containing the symmetric difference of ALLIDs with the original key. Computing it on the fly involves ALLIDs, which puts it right in to the unindexed category. Maintaining such a collection of indexes looks like a significant increase in processing time, I/O to the DB, and data size in the indexes. I'm guessing a bit of analysis on that may well show the cost is prohibitive. And in many cases any significant additional processing of this kind ought to be avoided. Maybe someone in ENG will chime in here with more details.
But if you already know all the other values you are looking for, can't you do a compound OR filter and get them all that way? It may still be unindexed if the candidate list grows over time to be more than your ALLIDs threshold. To be frank this query does not look like something that ought to be indexed. It grabs a pretty large number of your users. Your ALLIDS threshold already looks too high to me. If this is a reporting query of some kind, I'd recommend either parsing an LDIF backup (if you don't need realtime data) or using a utility/reporting server that you can bog down with long-running queries. If you are already using a utility server for this, you can examine whether the query really needs to be indexed at all. It may not make sense to try to index it.
As to why an operator would be available but not indexable, remember that the second major step of search processing examines each member of the candidate list and verifies a match on the filter. We routinely include filter components that we do not expect to assist in the creation of the candidate list, but we still want to be applied to the candidates during the filtering phase. For instance, consider the conventional filter (&(objectclass=person)(cn=XYZ)). In many Directories the objectclass filter component will evaluate to ALLIDs, and we want it to, because the cn component will have a much shorter list. We don't want the server to have to perform set intersection of a huge list of entryIDS (all the person objects) with the cn list. This is an example of how a too large ALLIDs threshold can hinder performance. We want the server to do the much easier intersection of ALLIDs with the short cn component list, because we know intersection of any subset of ALLIDs with ALLIDs is the subset. But we do want the server to check the candidates to make sure they really are person entries before returning them.