Taking the plunge - external drive and exclamation points

Question

Okay...
I have cried for help on this forum before about the somewhat common problem of iTunes 7 having a strange relationship with music on my external drive. The drive in question is a Lacie 250 GB Firewire drive.
A few months ago, out of the blue, one day I opened iTunes to find a sea of exclamation points. Though all the music I had added after a particular date was there, most of the music I had added prior to that was tagged with the exclamation. When I'd try to play one of these songs, iTunes would say it could not be found, though I could manually locate it and things would work fine. I tried a few fixes recommended on this forum by well-meaning but misguided people that resulted in a few sets of duplicate playlists but no fix to the problem. Then, one day, out of the blue again...it started working fine.
I don't even recall having restarted the computer. iTunes had just fixed the problem somehow.
Well, now the problem is back, and I've dealt with it for about a month. But I'm not convinced that it will just go away this time. So, even if I destroy my painstakingly cultivated hundreds of playlists, ratings, play history and comprehensive information (artwork, composer, etc.), I am determined to fix this problem without manually locating over 10,000 songs.
I am posting my attempts here as a record that will hopefully help those struggling with this problem in the future. Though I would welcome any and all comments or questions, I have frankly come to expect very little assistance or even the willingness to assist from this forum (that is to say, I imagine my unresolved question will languish for awhile as a monologue until I either fix the problem or die).
The first thing I will do (after doing a final backup, of course) is to determine whether it makes a difference if an updated version of my iTunes library file and .xml are in the Music folder (on my hard drive) or in my Musica folder (on the external drive). The latest edition of the file is currently on the external drive.
So, here we go...
Grant
Dual 1GHz G4, 1G RAM, 80G HD   Mac OS X (10.4.9)   iPod (1st Gen), iPod Shuffle (1st Gen), iPod Mini, External HD

Answer 1

Well, I've made quite a mess of things. So I have 14,000 songs. Most were added years ago. Looking at the date added in correlation with the exclamation points, it seems as though iTunes forgot the location of most of the songs added before 5/9/06. Mind you, iTunes worked fine (with the exception of doing this whole thing once before) until about a month ago. So, in May 2007, iTunes forgot most of what happened before 5/9/06.
(For reference purposes, the gap seems to exist between two songs I downloaded. "Help Me, Ronda" by the Beach Boys is the last song with an exclamation point in the date added list (see two exceptions below). "Proud Mary" by Ike & Tina Turner, on the other hand, is exclamation point free and only a couple of songs added after "Proud Mary" have exclamation points. They are:
Freedom, by Richie Havens, added 6/3/06.
Slow Free Bird, by Geesekey, added 12/17/06, a song I made in Garageband.
Now, don't let me give you the impression that there is a solid wall of exclamation points up to "Help Me, Rhonda." Maybe about every 10/20 songs, there is a song with no exclamation point that plays fine. Following Jase's lead before (his advice to another user), I have investigated some of these songs that work fine, but can't find any reason why these work and others don't (any difference in the iTunes Library.xml, files created with/without track number, etc.).
So, here's what happened in the past couple of days: I dragged my "iTunes Music" folder (located in my "Musica" folder on the external drive) into the iTunes source list). Most everything was duplicated, with the new songs playing just fine but lacking ratings and play history. Also, the new songs were not included on any playlists.
What I did next was poorly executed. I sorted iTunes by the Date Added and selected the first few thousand (roughly, A-J; seeing them according to the Date Added reveal that they weren't added in exactly a linear fashion). I hit the delete button and selected "Remove", but I then told iTunes to "Move to Trash." What I should have done, I imagine, is told iTunes to "Keep Files". So, my A-J songs are now deleted off the External Drive and gone (though I have backup DVDs generated by the iTunes "Back up to Disc" feature).
What should I do next? is the pressing question. Should I rip everything out and start from scratch with my backup discs? I guess the question I would have with that approach is this: if I restore from backup discs made three months ago, will I still have all my playlists, ratings and play history as they were when I made the backup? If that's the case, that'd be fine with me...I just want to make sure that's what will happen. Also, I have a backup of my iTunes Library file before I've done all this crazy stuff of the past few days, so if that enters into the equation, let me know.
Otherwise, let me address your comments, Jase.
Anyway, here's a question. Can you export your
playlists, even though they're made up of songs with
exclamation marks? If so, you could try exporting one
of them as an XML file. Then import it--you'll
probably wind up with duplicates (playlists, not
songs). Does the newly imported playlist play, or
does it still have the exclamation mark problem? I
wouldn't know what to expect, but it might be worth a
try.
My playlists themselves are all still the same as they were (though there are exclamation marks all over the place). From what I can tell, they look fine when exported to an .xml (the music folder seems to have the correct path). When I import it, it's the same deal: exclamation points at the same places.
If your playlists are exportable, you might want to
export them to XML files, just so you'll have those
little databases of them. Then, if worse came to
worse and you had to give up on your current Library
file and reload your Music folder into a new library,
at least there's a chance you could import your
playlists and not have to build them again from
scratch (assuming nothing has changed in terms of the
organization of your Music folder).
I'm not willing to export every playlist, because I have hundreds and it would just take too long. Is there a way to export all or a lot of playlists at once? Anyway, I figured that my playlist data was stored somehow in the iTunes Library file, so I felt I had a backup already. I could be wrong about this, though, as I really don't understand much about the iTunes Library file and .xml and the like.
Can you drag songs that have exclamation marks into a
playlist? Somehow, I doubt it. But if you can, you
could use new playlists to make a record of some
information that you might want to save if you had to
start all over again with a new Library.
Believe it or not, I can add songs with an exclamation point into playlists and they're added just fine (though, they still have the exclamation point). How this works is: I can't drag and drop them into a playlist. However, I can control-click and select "Add to Playlist" and it works. There's another mystery for you.
Have you looked at your iTunes Library.xml file --
specifically at the location information for
individual songs -- to see if the location
information exactly matches the information you see
in the Finder? I have found mismatches when
investigating some exclamation marked files. I didn't
know how that happened, and I didn't know what to
about it, but it was intriguing to see evidence of
some iTunes mischief.
Is the iTunes feature to start music file names with
the track number turned on? To my mind, fewer missing
song file problems occur when this is on.
Yes and yes. Based on what you told others, I have looked at the .xml of some of the problem songs and the working ones and could find no difference, but I didn't really know what I was supposed to be looking for. I also use the Add Track Number feature and always have.
If you still have the 28,000 songs in your Library
and want to go back to the 14,000, you could sort all
your songs by Date Added and delete the ones you
added on June 21. Or, if you saved a copy of your
iTunes Library file before you dragged your iTunes
Music folder into the library, you could just throw
out the more recent Library file and reinstitute the
back-up copy.
I probably should have done the latter (just restore the backup of the iTunes Library file). Oh well...
Some advice for future experiments. Try them with a
small sample if possible unless you're absolutely
sure they will work the way you want them to. For
example, drag one album in your Music folder into
iTunes rather than the whole Music folder.
This is a great idea. I wish I would have checked my sample more closely or done more tests before I plunged off the cliff.
Back up your iTunes Library file, as well as your
Music folder. (You might already do this.)
Check.
Those are a few things to think about. They probably
will not be helpful. Like I said, I don't think there
are clear answers to this problem. The best I can
wish for you is that iTunes will magically fix the
problem, like it did once before. Then, maybe there
are some preventive measures you can take.
Since iTunes did fix it once before, I would not
throw your current iTunes Library file away, no
matter what. If you decide to start a new library so
that you can actually listen to your music, rename
this Library file and keep it around, since it has a
lot of your metadata in it. You can try it every now
and then and see if iTunes spontaneously revives it.
Okay, I will post what I do next, provided that I don't hear any compelling advice and make my own decision (though I should be even more gun-shy after my recent blunders).
Grant