Unlogged Missing Photos After Import From Aperture

Question

Hi!
I have just made the switch from Aperture to Lightroom, and have use the 1.1 version of the Aperture import plugin.
In my Aperture Library I have, according to the Library -> Photos: 11105 Photos, however after importing to Lightroom, I have only 10967 photos. I have checked the import log, and there were 4 items which failed to import - 3 were .mpo files (panoramas from an xPeria) and 1 was a .gif file. This leaves a deficit of 133 photos that I can't account for.
Is there any way to compare the aperture library to the lightroom library to see what is missing?

Answer 1

*WARNING* Once agin, this is a VERY long post! And this contains not only SQL, but heaps of command line fun!
TLDR Summary: Aperture is storing duplicates on disk (and referencing them in the DB) but hiding them in the GUI. Exactly how it does this, I'm not sure yet. And how to clean it up, I'm not sure either. But if you would like to know how I proved it, read on!
An update on handling metadata exported from Aperture. Once you have a file, if you try to view it in the terminal, perhaps like this:
$ less ApertureMetadataExtendedExport.txt
"ApertureMetadataExtendedExport.txt" may be a binary file.  See it anyway?
you will get that error. Turns out I was wrong, it's not (only?) due to the size of the file / line length; it's actually the file type Aperture creates:
$ file ApertureMetadataExtendedExport.txt
ApertureMetadataExtendedExport.txt: Little-endian UTF-16 Unicode text, with very long lines
The key bit being "Little-endian UTF-16", that is what is causing the shell to think it's binary. The little endian is not surprising, after all it's an X86_64 platform. The UTF-16 though is not able to be handled by the shell. So it has to be converted. There are command line utils, but Text Wrangler does the job nicely.
After conversion (to Unicode UTF-8):
$ file ApertureMetadataExtendedExport.txt
ApertureMetadataExtendedExport.txt: ASCII text, with very long lines
and
$ less ApertureMetadataExtendedExport.txt
Version Name    Title   Urgency Categories      Suppl. Categories       Keywords        Instructions    Date Created    Contact Creator Contact Job Title       City    State/Province  Country Job Identifier  Headline        Provider        Source  Copyright Notice        Caption Caption Writer  Rating  IPTC Subject Code       Usage Terms     Intellectual Genre      IPTC Scene      Location        ISO Country Code        Contact Address Contact City    Contact State/Providence        Contact Postal Code     Contact Country Contact Phone   Contact Email   Contact Website Label   Latitude        Longitude       Altitude        AltitudeRef
So, there you have it! That's what you have access to when exporting the metadata. Helpful? Well, at first glance I didn't think so - as the "Version Name" field is just "IMG_2104", no extension, no path etc. So if we have multiple images called "IMG_2104" we can't tell them apart (unless you have a few other fields to look at - and even then just comparing to the File System entries wouldn't be possible). But! In my last post, I mentioned that the Aperture SQLite DB (Library.apdb, the RKMasters table in particular) contained 11130 entries, and if you looked at the Schema, you would have noticed that there was a column called "originalVersionName" which should match! So, in theory, I can now create a small script to compare metadata with database and find my missing 25 files!
First of all, I need to add that, when exporting metadata in Aperture, you need to select all the photos! ... and it will take some time! In my case TextWrangler managed to handle the 11108 line file without any problems. And even better, after converting, I was able to view the file with less. This is a BIG step on my last attempt.
At this point it is worth pointing out that the file is tab-delimited (csv would be easier, of course) but we should be able to work with it anyway.
To extract the version name (first column) we can use awk:
$ cat ApertureMetadataExtendedExport.txt | awk -F'\t' '{print $1}' > ApertureMetadataVersionNames.txt
and we can compare the line counts of both input and output to ensure we got everything:
$ wc -l ApertureMetadataExtendedExport.txt
   11106 ApertureMetadataExtendedExport.txt
$ wc -l ApertureMetadataVersionNames.txt
   11106 ApertureMetadataVersionNames.txt
So far, so good! You might have noticed that the line count is 11106, not 11105, the input file has the header as I printed earlier. So we need to remove the first line. I just use vi for that.
Lastly, the file needs to be sorted, so we can ensure we are looking in the same order when comparing the metadata version names with the DB version names.
$ cat ApertureMetadataVersionNames.txt | sort > ApertureMetadataVersionNamesSorted.txt
To get the Version Names from the DB, fire up sqlite3:
$ sqlite3 Library.apdb
sqlite> .output ApertureDBMasterVersionNames.txt
sqlite> select originalVersionName from RKMaster;
sqlite> .exit
Checking the line count in the DB Output:
$ wc -l ApertureDBMasterVersionNames.txt
   11130 ApertureDBMasterVersionNames.txt
Brilliant! 11130 lines as expected. Then sort as we did before:
$ cat ApertureDBMasterVersionNames.txt | sort > ApertureDBMasterVersionNamesSorted.txt
So, now, in theory, running a diff on both files, should reveal the 25 missing files.... I must admit, I'm rather excited at this point!
$ diff ApertureDBMasterVersionNamesSorted.txt ApertureMetadataVersionNamesSorted.txt
IT WORKED! The output is a list of changes you need to make to the second input file to make it look the same as the first. Essentially, this will (in my case) show the Version Names that are missing in Aperture that are present on the File System.
So, a line like this:
1280,1281d1279
< IMG_0144
< IMG_0144
basically just means, that there are IMG_0144 appears twice more in the DB than in the Metadata. Note: this is specific for the way I ordered the input files to diff; although you will get the same basic output if you reversed the input files to diff, the interpretation is obviously reversed) as shown here: (note in the first output, we have 'd' for deleted, and in the second output it's 'a' for added)
1279a1280,1281
> IMG_0144
> IMG_0144
In anycase, looking through my output and counting, I indeed have 25 images to investigate. The problem here is we just have a version name, fortunately in my output, most are unique with just a couple of duplicates. This leads me to believe that my "missing" files are actually Aperture handling duplicates (though why it's hiding them I'm not sure). I could, in my DB dump look at the path etc as well and that might help, but as it's just 25 cases, I will instead get a FS dump, and grep for the version name. This will give me all the files on the FS that match. I can then look at each and see what's happening.
Dumping a list of master files from the FS: (execute from within the Masters directory of your Aperture library)
$ find . -type f > ApertureFSMasters.txt
This will be a list including path (relative to Master) which is exactly what we want. Then grep for each version name. For example:
$ grep IMG_0144 ApertureFSMasters.txt
./2014/04/11/20140411-222634/IMG_0144.JPG
./2014/04/23/20140423-070845/IMG_0144 (1).jpg
./2014/04/23/20140423-070845/IMG_0144.jpg
./2014/06/28/20140628-215220/IMG_0144.JPG
Here is a solid bit of information! On the FS i have 4 files called IMG_0144, yet if I look in the GUI (or metadata dump) I only have 2.
$ grep IMG_0144 ApertureMetadataVersionNamesSorted.txt
IMG_0144
IMG_0144
So, there is two files already!
The path preceding the image in the FS dump, is the date of import. So I can see that two were imported at the same time, and two separately. The two that show up in the GUI have import sessions of 2014-06-28 @ 09:52:20 PM and 2014-04-11 @ 10:26:34 PM. That means that the first and last are the two files that show in the GUI, the middle two do not.... Why are they not in the GUI (yet are in the DB) and why do they have the exact same import date/time? I have no answer to that yet!
I used open <filename> from the terminal prompt to view each file, and 3 out of my 4 are identical, and the fourth different.
So, lastly, with a little command line fu, we can make a useful script to tell us what we want to know:
#! /bin/bash
grep $1 ApertureFSMasters.txt | sed 's|\.|Masters|' | awk '{print "<full path to Aperture Library folder>"$0}' | \
while read line; do
  openssl sha1 "$line"
done
replace the <full path to Aperture Library folder> with the full path to you Aperture Library Folder, perhaps /volumes/some_disk_name/some_username/Pictures/.... etc. Then chmod 755 the script, and execute ./<scriptname> <version name> so something like
$ ./calculateSHA.sh IMG_0144
What we're doing here is taking in the version name we want to find (for example IMG_0144), and we are looking for it in the FS dump list. Remember that file contains image files relative to the Aperture Library Master path, which look something like "./YYYY/MM/DD/YYYYMMDD-HHMMSS/<FILENAME>" - we use sed to replace the "./" part with "Masters". Then we pipe it to awk, and insert the full path to aperture before the file name, the end result is a line which contains the absolute path to an image. There are several other ways to solve this, such as generating the FS dump from the root dir. You could also combine the awk into the sed (or the sed into the awk).. but this works. Each line is then passed, one at a time, to the openssl program to calculate the sha-1 checksum for that image. If a SHA-1 matches, then those files are identical (yes, there is a small chance of a collision in SHA-1, but it's unlikely!).
So, at the end of all this, you can see exactly whats going on. And in my case, Aperture is storing duplicates on disk, and not showing them in the GUI. To be honest, I don't actually know how to clean this up now! So if anyone has any ideas. Please let me know I can't just delete the files on disk, as they are referenced in the DB. I guess it doesn't make too much difference, but my personality requires me to clean this up (at the very least to provide closure on this thread).
The final point to make here is that, since Lightroom also has 11126 images (11130 less 4 non-compatible files). Then it has taken all the duplicates in the import.
Well, that was a fun journey, and I learned a lot about Aperture in the process. And yes, I know this is a Lightroom forum and maybe this info would be better on the Aperture forum, I will probably update it there too. But there is some tie back to the Lightroom importer to let people know whats happening internally. (I guess I should update my earlier post, where I assumed the Lightroom Aperture import plugin was using the FS only, it *could* be using the DB as well (and probably is, so it can get more metadata))
UPDATE: I jumped the gun a bit here, and based my conclusion on limited data. I have finished calculating the SHA-1 for all my missing versions. As well as comparing the counts in the GUI, to the counts in the FS. For the most part, where the GUI count is lower than the FS count, there is a clear duplicate (two files with the same SHA-1). However I have a few cases, where the FS count is higher, and all the images on disk have different SHA-1's! Picking one at random from my list; I have 3 images in the GUI called IMG_0843. On disk I have 4 files all with different SHA-1's. Viewing the actual images, 2 look the same, and the other 2 are different. So that matches 3 "unique" images.
Using Preview to inspect the exif data for the images which look the same:
image 1:
Pixel X Dimension: 1 536
Pixel Y Dimension: 2 048
image 2:
Pixel X Dimension: 3 264
Pixel Y Dimension: 2 448
(image 2 also has an extra Regions dictionary in the exit)
So! These two images are not identical (we knew that from the SHA-1), but they are similar (content is the same - resolution is the same) yet Aperture is treating these as duplicates it seems.. that's not good! does this mean that if I resize an image for the web, and keep both, that Aperture won't show me both? (at least it keeps both on disk though, I guess...)
The resolution of image 1, is suspiciously like the resolutions that were uploaded to (the original version of) iCloud Photos on the iPhone (one of the reasons I never used it). And indeed, the photo I chose at random here, is one that I have in an iCloud stored album (I have created a screensaver synced to iCloud, to use on my various Mac's and AppleTVs). Examining the data for the cloud version of the image, shows the resolution to be 1536x2048. The screensaver contains 22 images - I theorised earlier that these might be the missing images, perhaps I was right after all? Yet another avenue to explore.
Ok. I dumped the screensaver metadata, converted it to UTF-8, grabbed the version names, and sorted them (just like before). Then compared them to the output of the diff command. Yep! the 22 screensaver images match to 22 / 25 missing images. The other 3, appear to be exact duplicates (same SHA-1) of images already in the library. That almost solves it! So then, can I conclude that Lightroom has imported my iCloud Screensaver as normal photos of lower res? In which case, it would likely do it for any shared photo source in Aperture, and perhaps it would be wise to turn that feature off before importing to Lightroom?