Automatic Filing With Rules - Long

Question

This is a long post, sorry, but I think the details might be worthwhile for someone who might be like me and wants to automate their mail processing as much as possible. I do have a question below about how I might better configure some rules, and about whether I might be able to streamline my workflow.
I spent all day yesterday and most of the day today reorganizing my mailboxes, rules, smart mailboxes, etc., to clean up all my mail. I have messages dating back to 1999, and since I basically just color coded everything with rules, rather than filing messages in mailboxes, my inbox and sent messages were starting to approach the 2GB limit. Some of the earlier messages came straight from Claris Emailer (talk about old school), and then through all versions of OS X Mail from 10.0 to the current 10.4.9. It's easy enough to move stuff from my inbox into a new mailbox, but not quite as much so for sent messages. It made searching more difficult, and it is still a manual process.
Because I have so many messages (about 10,000 in total), I didn't want to have to manually go through everything to move them into separate organized mailboxes. So I created a whole new set of rules to make the filing as automatic as possible, and to keep it relatively automatic in the future. Of course rules can't catch everything, but doing as much as possible automatically, then letting me sort through what's left is certainly preferable. Unfortunately, it's taken so long to figure out because of Mail's completely inadequate rules for filtering and smart mailboxes. I poured through this forum looking for solutions to the various roadblocks I hit along the way, and I thank all of you who are very knowledgeable and have posted helpful replies to other people asking questions. Below is the best solution I've been able to come up with, and I'm certainly open to any suggestions to make the rules less risky and streamlined. I realize that AppleScript could probably make a lot of these things easier, or some of the shareware programs out there, but while I consider myself a very adept power user, I can't write scripts, and I prefer to keep add-ons to a minimum.
The first change to my account settings and such was to enable the "Automatically CC myself" option. This is the only viable way I can find to process my sent messages without having to manually apply rules after the fact. I am aware of Andreas Amann's "Filter Sent Messages" AppleScript. I tried it out, but it was constantly choking on all my sent messages. I suppose it would work ok now that I have my sent mailboxes cleaned out, but I think I prefer the CC option because it assures that I have copies of all my sent messages at both my home and work computers. I also installed MailEnhancer to make sure that Mail's dock badge counts all unread messages in any mailbox, not just the Inbox. Now the fun stuff begins.
First of all, I want to state that since my goal here is to stay away from the mailbox size limit (and also to speed up day-to-day use of Mail), smart mailboxes aren't really an option. I need my messages sorted to "real" mailboxes for their long-term storage. Also, simply dumping mail into yearly archives can be a big hassle too, because I have several projects/mailboxes/whatever that may not have a ton of messages and shouldn't really be split up arbitrarily by year. Thus, the main focus here is on rules. I have two primary e-mail accounts, home and work, that I check regularly from both locations. I have my home mail broken down into categories like bicycling notices, family, apartment landlord, photography side business, and streetcar/railroad website. I also set up mailboxes for miscellaneous stuff that doesn't fit into those categories. For work, I have several mailboxes for project specific correspondence, office-wide e-mails and announcements, office computers, our website, and again miscellaneous.
Here's the other big paradigm shift. All these mailboxes contain both sent and received mail. This is especially valuable for the project work, because you can quickly look at the list of messages and follow the back-and-forth conversation. I originally started out having "sent" and "received" mailboxes for each project and other category that I've already listed, but when they're open it triples the size of the directory tree. Since there doesn't appear to be any way to make a mailbox/folder display all the messages contained inside its subfolders, I don't see much point in making a whole bunch of subfolders for that since I'd constantly have to open and close folders to get at anything. Also, with a more flat structure, I can quickly and easily see what unread messages are for what projects, and that also tells me if the rules are working well.
So how do I differentiate sent versus received easily? The first rules I set up are to color the background of all messages that come into my home account. This would apply to all messages that come in to that account, whether it's one of my sent message CC's or not. The next rule catches the sent messages by looking to see if the account and the from address match. In that case it colors the message subject text, marks the message as read, and also flags it. Flagging is necessary to create smart mailboxes down the line that can differentiate between sent and received messages, since there's no facility for using the color of a message as a condition in a rule or smart mailbox. These two rules are repeated with different colors for my work account.
Since none of the rules have moved any messages yet, further rules are free to process them. Basically I have one rule for each mailbox with various amounts of "any" criteria. For most situations it's pretty straightforward, but I have run into some difficulties with Mail's anemic rules. For example, for my family mailbox, I set a condition where "Sender is member of Group" "Family" that I set up in Address book. So far so good, but since I want this to grab sent messages to family as well, I need an "Any Recipient is member of Group" condition, which doesn't exist. The only way I can see to do this is to manually add all my family members' individual e-mail addresses as separate "any" conditions. That would make the rule look like this:
If "any" of the following conditions are met:
"Sender is member of Group" "Family"
"Any Recipient" "Contains" "[email protected]"
"Any Recipient" "Contains" "[email protected]"
"Any Recipient" "Contains" "[email protected]"
"Any Recipient" "Contains" "[email protected]"
etc.
Perform the following actions:
"Move Message" to mailbox: "Family"
Of course, what it SHOULD be is this:
If "any" of the following conditions are met:
"Sender is member of Group" "Family"
"Any Recipient is member of Group" "Family"
Perform the following actions:
"Move Message" to mailbox: "Family"
This one is tripping me up a bit, and I'm not sure how to use address book groups effectively for sent messages. This is also an issue for some of my work project e-mails, since conceivably I'd be sending messages to and receiving messages from the same group of people. For larger projects, my list could grow enough to where I can't add anymore conditions to the rule, which I know is a well-discussed bug (or lack of good coding anyway). Nevertheless, it seems that in order to capture a message with a specific e-mail address, I need to have two conditions for each address, one "from" and one "any recipient". That kinda stinks, and I'd like to find a better way to do it. Any ideas?
Right now my last rules are to clean up any messages left over that the other rules didn't catch. It makes the color a bit darker and moves the messages to the appropriate "other" mailbox for that account. This ensures that my inbox is always empty. It also keeps my sent mailboxes cleaned out as well. Since I will get CC's of all my sent messages, I can just change my account preferences to delete sent messages after a day or a week.
That's it for the heavy moving and lifting. With a few of my extra accounts that don't need any special filtering and moving around, I was able to accomplish this with about 30 rules. There's 2 rules per account at the top of the list to color and flag incoming and outgoing messages, and two rules per account at the bottom of the list to file any messages that don't meet the criteria for any of the other mailboxes. All the actual filing away into specific mailboxes happens in the rules in the middle of the list. What I really need to do though is figure out a way to simplify some of the conditions in those filing rules, like the family issue I mentioned above.
I have set up a few smart mailboxes to display all my unread messages, recent received mail, recent sent mail, all recent mail, all received mail, all sent mail, and all mail. The "All Mail" mailbox's criteria are just that the message is not in one of my junk mail folders. This is my fail-safe, since even if something is mis-categorized or even unfiltered, I will still see it here. "All Sent Mail" collects all messages that are flagged. Flagging is the only way I can find to view just sent messages without having separate "sent" subfolders in all mailboxes. A benefit to doing it this way is that if I add or change any of my hard mailboxes, I don't have to edit the smart mailbox to keep it current. I can hid the flag column and it won't reappear by itself, so I don't have to see the flag. "All Received Mail" is the tricky one. As mentioned elsewhere on the forum, you can't set up a smart mailbox to get messages that are NOT in another smart mailbox. This rule needs to include all messages that are not in the "All Sent Mail" smart mailbox or any of my junk folders. I tried the trick at first where you pick "Message is in Mailbox" and then select your smart mailbox, then go back and change the criteria to "Message is not in Mailbox". When I'd close the rule though, the smart mailbox I chose in the pull-down menu would change to a different one, and the logic would fail. I got it to work by trashing my smart mailboxes .plist file and just remaking them. That did the trick. The rest of my smart mailboxes just add a time limit to their list of criteria to keep the list shorter, and the unread smart mailbox is self-explanatory.
So what these smart mailboxes do is they let me see anything from just my unread messages to every single message I've ever sent or received in one window. They're all colored accurately and sortable by sent versus received. If I keep my "Recent Received Mail" mailbox as the active one, it looks just like my inbox did before I changed all this stuff around. The only difference is that this particular mailbox automatically displays only messages from the last 3 months, and I don't have to manually clean it up every so often. This makes opening mail itself and navigating recent messages quicker. In the end, I can still see the messages the same way I always have been, but since they're all stored in different hard mailboxes I'll be able to avoid the 2GB limit by keeping everything well categorized. Nevertheless, the logic required to make this all work is quite cumbersome, and I worry about it not working correctly down the road as Mail is updated. I'd appreciate any thoughts you all have on making this work better, and about my family rule problem.
Thanks all

Answer 1

> As to marking sent messages versus received, if you didn't do that,
how would you be able to tell them apart easily?
It’s the need to tell them apart that I don’t understand...
You'd want to be able to tell them apart for the same reasons
you'd want to tell anything apart.
But I can already easily tell them apart without having to tag, colorize, or store them in separate mailboxes, so why waste an attribute of a message to store redundant information? It’d be better to reserve that attribute for storing information that cannot be made readily available by other means, e.g. subjective properties such as the importance of the message that you mention.
One instance I can site is if someone asks "did you send me such-and-such?"
In that case, I can, for example, type the first letters of the name or address of that person in the search box, then click All Mailboxes and To in the banner that appears between the toolbar and the message list. Mail shows me what I’m looking for without me having to leave the mailbox I was looking at nor wonder on which specific mailbox are those messages actually stored. Having all those messages tagged or colorized the same way would be useless or even distracting/annoying.
Unless you have a specific mailbox for storing only the correspondence with that person AND you can easily locate and select that mailbox amongst all the other mailboxes, I don’t see how could it be more convenient looking through a mailbox where messages can be told apart based on whether they’re sent or received but not on who sent/received them...