"Date Modified" for all files being changed if "Automatically write to XMP" is on

Question

Recently upgraded from LR3 (v3.4.1) from LR2 on OSX 10.6.8 and have always had Catalog Settings > Automatically write changes into XMP turned on.
When browsing existing JPG files in my Library (no Develop changes, no keywording, no Presets, no Import), LR3 is writing to disk — i.e., when I look at files in Finder, almost every viewed file’s “Date Modified” is being set to today’s date and time. (It actually creates a .swp file, then changes it's name back to .jpg)
This is really bad, as it's making it impossible for me to use Finder to figure out when I last worked with a file, it is triggering needless Time Machine and Backblaze backups, and unnecessarily churning my disk.
If I turn off "Automatically write..." this behavior stops. Per David Marx at thelightroomlab.com, I tried turning off this preference, manually doing a "Save Metadata to File" for all files, letting that complete, then turning the preference back on. This does not solve the problem.
Per a suggestion at photoshop.com, I used ExifTool to see what changes LR was writing to a sample file; from the diff below, you can see that LR is adding a bunch of new fields as well as moving other fields around. But my point is that LR3 should never overwrite a file on disk if all I am doing is browsing thru them.
Is anyone else seeing this? Any ideas would be greatly appreciated!
-- David
diff Exif5609_original Exif5609_update
2c2
< FileName: DSC_5609_original.JPG
> FileName: DSC_5609_update.JPG
5,6c5,6
< FileModifyDate: 2009:11:27 21:32:54-08:00
< FilePermissions: rwxr-xr-x
> FileModifyDate: 2011:08:07 22:06:47-07:00
> FilePermissions: rw-r--r--
27a28,29
> ShutterSpeedValue: 1/200
> ApertureValue: 7.1
55d56
< SerialNumber: 3209521
75d75
< Lens: 18-200mm f/3.5-5.6
185,188d184
< UserComment:
< SubSecTime: 00
< SubSecTimeOriginal: 00
< SubSecTimeDigitized: 00
211a208,299
> XMPToolkit: Adobe XMP Core 5.2-c004 1.136881, 2010/06/10-18:11:35
> CreatorTool: Ver.1.00
> MetadataDate: 2011:08:07 22:06:47-07:00
> SerialNumber: 3209521
> LensInfo: 18-200mm f/3.5-5.6
> Lens: 18.0-200.0 mm f/3.5-5.6
> ImageNumber: 26634
> RawFileName: DSC_5609.JPG
> SavedSettingsName: Import
> SavedSettingsType: Snapshot
> SavedSettingsParametersVersion: 6.4.1
> SavedSettingsParametersProcessVersion: 5.0
> SavedSettingsParametersWhiteBalance: As Shot
> SavedSettingsParametersIncrementalTemperature: 0
> SavedSettingsParametersIncrementalTint: 0
> SavedSettingsParametersExposure: 0.00
> SavedSettingsParametersShadows: 0
> SavedSettingsParametersBrightness: 0
> SavedSettingsParametersContrast: 0
> SavedSettingsParametersSaturation: 0
> SavedSettingsParametersSharpness: 0
> SavedSettingsParametersLuminanceSmoothing: 0
> SavedSettingsParametersColorNoiseReduction: 0
> SavedSettingsParametersChromaticAberrationR: 0
> SavedSettingsParametersChromaticAberrationB: 0
> SavedSettingsParametersVignetteAmount: 0
> SavedSettingsParametersShadowTint: 0
> SavedSettingsParametersRedHue: 0
> SavedSettingsParametersRedSaturation: 0
> SavedSettingsParametersGreenHue: 0
> SavedSettingsParametersGreenSaturation: 0
> SavedSettingsParametersBlueHue: 0
> SavedSettingsParametersBlueSaturation: 0
> SavedSettingsParametersFillLight: 0
> SavedSettingsParametersVibrance: 0
> SavedSettingsParametersHighlightRecovery: 0
> SavedSettingsParametersClarity: 0
> SavedSettingsParametersDefringe: 0
> SavedSettingsParametersHueAdjustmentRed: 0
> SavedSettingsParametersHueAdjustmentOrange: 0
> SavedSettingsParametersHueAdjustmentYellow: 0
> SavedSettingsParametersHueAdjustmentGreen: 0
> SavedSettingsParametersHueAdjustmentAqua: 0
> SavedSettingsParametersHueAdjustmentBlue: 0
> SavedSettingsParametersHueAdjustmentPurple: 0
> SavedSettingsParametersHueAdjustmentMagenta: 0
> SavedSettingsParametersSaturationAdjustmentRed: 0
> SavedSettingsParametersSaturationAdjustmentOrange: 0
> SavedSettingsParametersSaturationAdjustmentYellow: 0
> SavedSettingsParametersSaturationAdjustmentGreen: 0
> SavedSettingsParametersSaturationAdjustmentAqua: 0
> SavedSettingsParametersSaturationAdjustmentBlue: 0
> SavedSettingsParametersSaturationAdjustmentPurple: 0
> SavedSettingsParametersSaturationAdjustmentMagenta: 0
> SavedSettingsParametersLuminanceAdjustmentRed: 0
> SavedSettingsParametersLuminanceAdjustmentOrange: 0
> SavedSettingsParametersLuminanceAdjustmentYellow: 0
> SavedSettingsParametersLuminanceAdjustmentGreen: 0
> SavedSettingsParametersLuminanceAdjustmentAqua: 0
> SavedSettingsParametersLuminanceAdjustmentBlue: 0
> SavedSettingsParametersLuminanceAdjustmentPurple: 0
> SavedSettingsParametersLuminanceAdjustmentMagenta: 0
> SavedSettingsParametersSplitToningShadowHue: 0
> SavedSettingsParametersSplitToningShadowSaturation: 0
> SavedSettingsParametersSplitToningHighlightHue: 0
> SavedSettingsParametersSplitToningHighlightSaturation: 0
> SavedSettingsParametersSplitToningBalance: 0
> SavedSettingsParametersParametricShadows: 0
> SavedSettingsParametersParametricDarks: 0
> SavedSettingsParametersParametricLights: 0
> SavedSettingsParametersParametricHighlights: 0
> SavedSettingsParametersParametricShadowSplit: 25
> SavedSettingsParametersParametricMidtoneSplit: 50
> SavedSettingsParametersParametricHighlightSplit: 75
> SavedSettingsParametersSharpenRadius: +1.0
> SavedSettingsParametersSharpenDetail: 25
> SavedSettingsParametersSharpenEdgeMasking: 0
> SavedSettingsParametersPostCropVignetteAmount: 0
> SavedSettingsParametersGrainAmount: 0
> SavedSettingsParametersLensProfileEnable: 0
> SavedSettingsParametersLensManualDistortionAmount: 0
> SavedSettingsParametersPerspectiveVertical: 0
> SavedSettingsParametersPerspectiveHorizontal: 0
> SavedSettingsParametersPerspectiveRotate: 0.0
> SavedSettingsParametersPerspectiveScale: 100
> SavedSettingsParametersConvertToGrayscale: False
> SavedSettingsParametersToneCurveName: Linear
> SavedSettingsParametersCameraProfile: Embedded
> SavedSettingsParametersCameraProfileDigest: D6AF5AEA62557FCE88BC099788BBD3CC
> SavedSettingsParametersLensProfileSetup: LensDefaults
> SavedSettingsParametersToneCurve: 0, 0, 255, 255
> IPTCDigest: d41d8cd98f00b204e9800998ecf8427e
228,230d315
< SubSecCreateDate: 2009:11:27 21:32:54.00
< SubSecDateTimeOriginal: 2009:11:27 21:32:54.00
< SubSecModifyDate: 2009:11:27 21:32:54.00
http://feedback.photoshop.com/photoshop_family/topics/lr3_date_modified_for_all_files_bein g_updated_when_browsing_photos_if_catalog_settings_automatically_write_changes_into_xmp_is /replies/6313647

Answer 1

clvrmnky wrote:
davidpope007 wrote:
Then when LR3 loaded my old LR2 images into memory, it "dirtied" the in-memory copy of the file by adding in these new LR3 XMP fields. Then, because I had "Automatically write XMP" on, it said "I better write these changes to disk".
Yuck. As a former software engineer, this is very bad software engineering.
It should wait until the user dirties the file (via Develop, keywords, etc.) before presuming to add a bunch of metadata fields that are unique to the new version of LR3.
Well, I'm a current software developer, and this is, really, a perfectly reasonable thing to do. It is a reasonable trade-off for a convenient feature required by a small subset of users.
Yes, in most cases the in-memory copy should "never" be dirtied unless the user makes a gesture of some sort, but like I said earlier, this option (once set by the user) sets up the situation where this gesture becomes implicit. This is a clear trade-off for the sake of convenience. And if the XMP is out of date and needs to be updated en masse, so be it.
The fact is, there is no easy way around this. Do we save up /every/ dirty buffer somehow until you make a gesture that /might/ require the XMP to be up-to-date before acting on that gesture? Now we have to worry about unflushed buffers if something goes wrong and the app exits. Do we save the buffers to the DB? Now we have to block some calls to make another blocking call to flush some or all of those to DB, and then write some or all of it out to one or more files. In what order? What if there is a gesture to have X files with up-to-date XMP and some or all of those are in unflushed buffers, unflushed DB writes or we have to wait for the DB.
As you can see, this is a transactionality nightmare, and the easiest and safest thing to get what the user wants (i.e., up-to-date XMP for the purpose of talking to a third-party XMP aware app) is to simply update the sidecar or XMP block in an atomic manner using the correct file IO. The file will have to change at some point, so it may as well be now.
[Thanks to both of you for your detailed replies. I am aware of the need for tradeoffs so when you say the approach taken is quite reasonable, I do believe you. I also apologize in advance for the length of the following and am extremely aware of the time it must have taken you to compose the above replies, but I'm going to add a bit more, if only for my own piece of mind and in hopes of coming up with a solution for my workflow.]
From my naive point of view, I was expecting the answer to be simply "don't raise the XMPDirtyFlag upon reading in a file". Obviously if your architecture requires you to "upgrade to latest XMP format" upon read, and another part of the system auto-detects "out of date XMP", then it's going to write those changes to disk.
But it didn't need to be designed that way. LR obviously has mechanisms to know when a user has made a change to XMP so it is able to write XMP changes to disk only when necessary.
The promise (to me) of "Automatically Write XMP changes to disk" was to auto-save my changes, and not those made for any internal (i.e., XMP versioning related) changes.
Perhaps the premise is that it is LR3's job to update an individual file's XMP to the latest version so that other XMP-aware apps can make use of it? I would argue that those third-party XMP-aware apps already have to know how to deal with all prior versions of XMP, so LR3 should just leave well enough alone.
You asked if my problem with your approach was that it was "inelegant"; not at all, it is based on my own perception of what I need from my workflow, so let me describe that so maybe we can find a better way:
* Part of the appeal of LR to me is that it preserves my original file as it came off the memory card, allowing me to move to a different workflow/toolset in 2025 if I choose to do so
* However, with all of changes contained in a single database file, I'm concerned about rare (but possible) corruption, so to mitigate this risk, I let LR backup my database weekly and it's also backed up continuously in the cloud via Backblaze
* Even with backups of the database, there is still a chance that I could lose changes made to individual files (e.g., LR corrupts the DB and I have to go back to last week's DB)
* Thus the appeal of the "auto-write to XMP" flag -- that way critical changes (develop, crop, keywords) are saved on a per-file basis; I liked the "automatic" part of this (as opposed to a manual save) because then I don't have to teach others in my family how to manually save XMP changes
* A nice side-effect of this setting is now when I use Finder to find a file and double-click on it to edit it in Photoshop, all my develop changes are right there; (in other words, I like the flexibility of not having to fire up LR in order to just invoke PS from within it); also when I use Bridge I see all the keywords there
* So with LR2, I had gotten used to what I thought was the best of all worlds -- autosave of changes at the file level via XMP + raw negatives untouched (i.e., Date Modified == the date I took the picture); this allows me to use operating-system-level tools -- Finder -- to locate/search for files
* Now I upgrade to LR3 and I'm finally now understanding that a concept "XMP versioning" is going to result in changes to many, but not all my files. (That's something else that's annoying about this issue -- I open up the Grid and browse a folder of files, and only seemingly random ones I've cursored over seem to get written to disk -- if it's so urgent that LR3 update the XMP, then it should do it for all the files in the catalog or at least in that directory)
Here's a screenshot from Finder of what I see everytime I look at this folder:
* So now I have to assume that each new version of XMP and/or LR is going to touch my files on disk. Sigh.
* What I don't like about this is that it is ruining the promise of "untouched raw negative". Yes, the image data is untouched -- which I agree is most of the benefit; but the file has been touched.
* Perhaps you might empathize a bit more if you imagined that someone went thru all your source code or Word files and randomly changed the date to "today" because you upgraded compilers or moved to Word 2011.
I agree all of this would be solved by having an XMP sidecar file for JPGs, but you indicate that's not going to happen.
You've also alluded to the solution of "resetting the Date Modifed" to it's original value -- which I believe is what Finder does when you move or copy a file -- but that that is fraught with issues as well. I believe you when you say there are issues, but again the naive part of me wonders why that soultion would be so bad...
I just thought of another potential solution -- turning on Date Created in Finder -- but it turns out that's changed, too.
I am really at a loss as to what to do and would welcome your suggestions.
Thanks again and kind regards,
-- David