How I solved my Spotlight+ Mail indexing problem

Question

OK, so I thought I would post my solution to a stubborn indexing problem, for people who may have the same issue (and it does seem many people do).
Problem: Spotlight index was unreliable. In particular, Spotlight was terrible at indexing mail. It could not find the messages by content that it would find by searching the "from" field, nor could it find those reliably. Unusable.
What DID NOT work:
-- adding and removing any folder whatsoever in the Privacy preferences of Spotlight. The index would rebuild, but with the same problems.
-- doing this via terminal (e.g, using sudo mdutil -E /Volumes/MyHardDisk)
-- using Onyx
-- using Spotless
-- simply reinstalling the system
-- manually modifying the plist files inside the .spotlight folder to add and remove the folders that would not get indexed.
What DID work:
-- first, I reinstalled the system. That's not such a big deal. Upgrade up to 10.4.6
-- then, I authorized the root user (that's probably not necessary) (this, you do by using Netinfo manager)
-- Then, I deactivated the index (via terminal:
sudo mdutil -i off /Volumes/MyHardDisk
-- now the tricky part. This concerns Mail, in particular.
I have a very complex mail folder (tens of thousands of messages). I noticed that inside each mail folder, I had two things that were not supposed to be there
1. a copy of each folder.mbox which was empty, but with the same name without the extension .mbox. This is probably something that was created while importing mail from Panther to Tiger, for what I can tell.
2. Most importantly, a file named ".index.ready". This also, apparently, was an old legacy coming from Panther. I suspect this was the main problem for the lack of indexing in Mail.
Ok, so... I deleted these folders and files. If you have a complex structure of subfolders, you can delete the files with an Unix command by first cd to your mail folder, and then use
find . -name \.index.r* -ok rm {} \;
which will ask you confirmation before deleting each file.
It was faster to remove the empty subfolders by hand, as they all had different names.
-- Ok, at this point, I erased entirely the .Spotlight-V100 directory which stays at the root of your computer. You can do it with a rmdir command from the terminal or, as I did, you can login as root (remember, we activated root), use a utility that allows you to see hidden files (e.g., Tinkertool), and move it manually to the trash folder.
Then I emptied the trash. For superstition, I restarted, and let the index be built again (that took about 5 hours in my computer, for an HD with about 60 gb occupied on it).
Now everything seems to work perfectly fine. I don't know whether this procedure can be shortened, but I do know that nothing else worked for me. I thought I would post it here as I spent sooo much time to solve this problem, and furthermore, I found that none of the suggested solutions would work.
OK, your responsibility to do it, if you want to do it. Hope nobody needs to!
Best to you all,
l.
  Mac OS X (10.4.6)  

Answer 1

Thank you, thank you, thank you, Lucabo! Using ideas from your post, I also (at least for now) solved my Mail indexing problem. One day (not even after an upgrade) Mail just quit finding messages by content, even when I had open a message with that content!
What I did:
-- In System Preferences, told Spotlight to index only Mail (I just use grep from a Terminal window for everything else!)
-- Ensured Mail app wasn't running
-- Backed up my Library/Mail folder to an external disk
-- Turned off indexing via Terminal (after sudo sh), with
mdutil -i off /Volumes/MyHardDisk
-- Deleted old files in my mailboxes (inside the whatever.mbox folders):
.index.ready
mbox
tableofcontents
content_index
It was clear that mbox, tableofcontents, and content_index were left over from a previous version of mail - they hadn't been accessed since I upgraded.
Note that I didn't delete the whatever.mbox folders (just the plain mbox file inside)!
-- Deleted (in Terminal) the .Spotlight-V100 in the root directory
-- Restarted my Mac. The indexing started itself. Since I had it set up to index only Mail, it took about 3-ish hours to finish (I have 100s of 1000s of emails in thousands of folders).
After the indexing finished, all the test searches that I tried worked. From now on I will always be suspicious if search doesn't find anything!
Also, I just checked - using the Spotlight field on the menu bar now works better too. When the Mail search problem showed up, I tried using the Spotlight field and the Spotlight window, but they would just flash up a few items, remove the list, over and over, sometimes with the beachball, then finally settle to an empty list. Now the Spotlight field seems to work.
I infer that the Spotlight index gets corrupted, and that the Spotlight code doesn't notice that it's reading junk from the index...
BTW, about the same time that search quit working, some emails in my Inboxes got highlighted with a dark brown background color. Have no idea why. They aren't marked as Junk (the junk messages have the text in the usual lighter brown). But, when I select a message with the dark brown highlight, mark it as junk, then mark it as not junk, the dark brown highlight goes away. On some occasions, just quitting Mail and restarting makes the dark brown highlights go away - but not always. Sigh... at least it's not a show stopper, even if it is ugly!
Thanks again, Lucabo, for your post!
  Mac OS X (10.4.6)