Xmp sync problems--corrupted files?

Question

I have been having lots of problems getting my xmp metadata sorted out. I am a new user to Lightroom, but I think I have figured out most of what I need successfully.
I use a variety of programs to edit the xmp data (mostly Lightroom and exiftool right now), so I know the importance of the "Automatically write XMP data to file setting" (ON in my case). After a few days (actually more like a week) of writing xmp data and then reading into (or synching folders in) Lightroom, I found a number of "metadata conflicts" in my catalog. Since these are basically impossible to fix for a large catalog (since there is no way to show the files that have conflicts), I started over and created a new catalog, and imported all 40,000+ images into it.
However, with my new catalog (with presumably no metadata conflicts), the first image that I tried to change the metadata on gave me a metadata conflict (presumably when Lightroom tried to save the xmp data after I changed it). This was a jpg file, so the metadata is being (or not being) written into the file. Reading the metadata from the file into Lightroom did not fix the problem. The Metadata Status field in the Lightroom Metadata panel says "Unknown" and clicking on the word "Unknown" brings up a panel saying "There was an unknown error while writing metadata to this photo. Retry?", and neither retrying the save nor importing the settings from disk resolves the conflict.
I think that the problem is that the jpeg is somehow corrupt, because if I save a copy in Photoshop, I can read and write the metadata in that copy. And, the copy has, I think, the same metadata as the original. (Of course, the image is probably imperceptibly degraded because I re-jpeg'd it).
There is no message in Console (I'm on Mac). (See PS)
I am worried that I have a number of files with this kind of problem, and I'd like to find the problem and fix it. Any ideas for a strategy to go about doing this? I guess I could "Save Metadata" for all files in my catalog (thus updating the file modified date for all 2 TB of data, and causing a massive backup issue), and then look through my catalog for those elusive exclamation points. But I'd like something better.
I have no idea how the jpeg could have gotten corrupted, but I do have an archived version that I could look at.
Any thoughts on next steps?
TIA
Alan
PS--I did find this in my console output hundreds of times:
8/10/11 2:40:00 PM    Adobe Lightroom 3[404]    KeywordUsageStatistics    ERROR    Bad popularity entry for 1937630: occurrences is negative: -1
I don't know if that is related, but it worries me.

Answer 1

After moving LR3.5 and catalogs to a new computer running Windows 7 and LR from an XP system, I suddenly had 6400 files with exclamation points--failure to synch metadata or similar warnings on every file.  If I thought I had one without the error, if I clicked on it--there error showed up.    It was very frustrating and I was wondering if I was going to have to keep my old computer forever.
After I searched for a couple of hours and found that there have been several discussions concerning failures to sync xmp sidecar files and failures to write metadata on this forum and others.  Some of the postings had angry rants against Adobe.  On my system, it was not Adobe that should be blamed, it was Microsoft:
On one of the Adobe threads, I noticed a post that the person had multiple external drives and the problem did not show itself on one drive.  The person said that changing the permissions on the other hard drives solved the problem.   I tried this and it worked for me.  I did not have this problem with this same drive on my XP system.  I will try to find that other forum and thank the person who gave me the solution.