Remove duplicates, keeping highest quality file.

I'm hoping someone can help me. Through recovering and manually rebuilding my library, I end up with multiple duplicates of multiple files. Maybe 15,000 duplicates out of about 37'000 files. I'm looking for a way via Osx (maybe Automator) or an application to remove multiple duplicates while keeping the highest quality image. This could be either the highest resolution or the largest file size. I've tried duplicate annihilation but that app does not have an option to accommodate this. I giess its important to note that the identical files could look like this:
Img-1234. 468kb
Img-1243 2 824kb
Img-1234 copy 2 1.4mb
2012-12-12 1.4mb
What I would want to achieve from these four copies of the exact same picture is to flag all but one of the 1.4mb files as duplicates.
Does anyone know of a way to do this other than manually?

Have you looked at their web site to see if it does what you want?
http://www.fatcatsoftware.com/iplm/Help/table%20of%20contents.html
LN

Similar Messages

Choosing to play highest quality file - can you see how?

I can't figure out how to do this using smart playlists, even nested ones. Perhaps the brain trust can help me figure this out.
I have a large library in which I have some songs encoded as both Apple Lossless and AAC. Other tracks are AAC only, and still others are MP3. Occasionally there are songs that are both AAC and MP3.
I'd like to have a playlist that contained only the highest quality version of each song. So if there is an Apple Lossless version and an AAC version, only the Apple Lossless version would be in the list.
I have made a smart playlist that contains all the Apple Lossless files, but I can't figure out how to filter out all the AAC versions, while still leaving in the AAC only songs.
It would seem that you have to be able to compare each song which you can't do with the smart playlist logic.
Thanks for any thoughts.
Jordan

Use the 'Less Than' or 'Is In the Range' then select relevant values. So for instance if you selected 'Is In the Range' and entered 128 - 320 the playlist will fill with all the files that fall in that range. Which will not include any Apple Lossless files. This will not sort out any MP3's though. I don't know how you can differentiate between MP3's and AAC. Unless all your AAC's are a higher bit rate than your MP3's?
Phil

Removing Duplicate photos and other files from Backup HD

Hi I have a macbook air with USB and Thunderbolt ports. I connect them to a 200GB Maxtor external harddrive via USB
The HD seems to have created duplicates of many of my files. How can i remove the duplictaes from my HD?
Thanks - Pete

Time Machine store backups of all data on your computer.
Changes to files are stored, so if required, you can revert to previous versions.
Once the disk fills with data, old backups are deleted to make room for the newer.
So Time Machine automatically manages the disk.

Exporting HD video to the highest quality file

Sorry if this has been covered but I cant find a setting that makes the quicktime file look as good as the video displayed in imovie. What is the best export to quicktime file setting?

Try Quicktime H.264, Best Quality (slider all the way to the right), current framerate current, other settings on Auto, size HD 1920x1080. Can't get much better than that.

Advice for keeping high quality files for archiving

I'm contemplating archiving here. I love reference files because they are pretty small in file size and don't compress. However, I have to worry that all my files will remain in the same exact place forever. I have found that is not always the case for whatever reason. Then there is self-contained reference files. A great idea...I don't have to worry about moved files but the file size is enormous. I do long videos (sometimes an hour or so) so file size is really important to me.
Is there anyway I can do a self-contained reference file but somehow reduce the file size or something? I know I'm throwing a bone here but I figured I would ask because my current scenario is not improving daily. I know I can put my hour or whatever to tape but that essentially is compressing it and I'm using valuable time. Any advice?

HI Lowsky, I been down this road some time back and my opinion is that if you are working in any workflows associated with CMFs (contemporary media format) in a tapeless workflow then storing your LONG TERM content as CMF's on spinning disk will eventually lead to misery and a hopeless state of anger and heartache. Spinning disk is not the long term answer.
refer here for lots of detail: I have post this detail in another forum at
http://www.reduser.net/forum/showthread.php?t=21284&page=6
*Archiving FCP Media Manager Projects, essence, import/transfer material.. etc*
Aside from many testimonials about DISK being faster (more accessible?) and convenient, storing content on spinning disk is a like as a child having all the toys you could imagine but only playing with your 3-4 favourites. (replace with any analogy). Simply you only need the stuff around on disk that you actually use a lot or currently.
I (and I'm not alone) don't trust and disk storage system unless you buy enterprise level disk arrays with some fault tolerance and smart disk array controllers AND that you have at least multiple instances of the content. I qualify ENTERPRISE disk systems - these at $USD5.00-$USD10.00/GB+. I mean those disk arrays from I,T. vendors, and M&E guys like Isolon, DDN, Apple, HP, Copan and their 40+ ilk.
The affordable stuff most of us is around $USD0.25-$USD0.75/GB - SATA and lowest end HBA's or simple controllers. Basically just spinning +power soaking heat dissipating buckets+ to put our content into. (USB | FW or the external JBOD disk enclosues with an HBA or higher FC disk arrays)
Most of these are enterprise disk storage systems that are out of the price range of many on this forum...
Devices like DROBO™ are cute but they are way way too slow (29MB/s when not busy consolidating itself ) and are very limited on their architecture for VIDEO work (useless). I gave mine away after a week to a photographer.
Alternatives like many sets of external and detachable and mobile spinning disks, leaving them spun down until use offer a great odea however unless the disk controller is savvy, you may as well have a bank of el cheapo Le Cie 1TB death disks and get on your knees and face the moon when you power them up.. (Good = luck with that!) mode=:rant
*Viable Archive Solutions for CMFs?*
After looking for viable solutions to by content storage problem after moving to a tapeless workflow with my HVX200 and P2, AND losing on two separate occasions a whole 1TB external disk (Le Cie).. losing most of the data of two drives (2 x 1TB over 4 months).. I have had it with disks!
For me, Disk storage should only be used like a +kitchen table+ for immediate and current and frequently accessed work. Like the +toy analogy+, hen not in use, put the toys away somewhere where they will not be damaged (the content objects) (like a large toy box or cupboard) that you can access with ease. (not the garden shed out the back nor the attic else it will never be used). This is workflow is simple enough..
My first thought for managing FCP media manager archives, P2 footage and final composites was to use BD's.. however the cost for the content that was wanted to archive was not cost effective. I had literally 100's of old video tapes, DV , HDV and ye old video8 rubbish that I needed to move to a CMF, all accessible via proxies and metadata (a la CATDV™ a mam from squarebox.co.uk). The restriction was COST of BD or DVD-DL media and worse the time of write and then access the material ESSENCE when I needed it.
I even looked for a while at many 100's of DVD-5's which in Hong Kong are as cheap as free beer coasters in a bar. I would buy them in canisters of 100's at a time. Simply these are cost effective using TOAST to make multivolumes self extracting archives, but are SOOOooooo slow to write and then access.
So I turned to looking at the economies and workflows of using LTO4 Ultrium data tape drive. I was quite surprised at the reasonable cost.
The cost of Ultrium LTO4 data tape media of to 800GB native for prices that are as cheap as $USD1.40/GB as pure media (the tape cartridge itself).It depend where you buy them and at what quantity. Of course there are the necessary offset/initial costs of the data tape drive AND the SAS HBA you will need for your mac. (LTO4 Ultrium @ $USD2700 and $USD150+ for SAS HBA). Much cheaper fo Ultrium LTO3 but lower data rates and capacities.
NExt year HP et al will have Ultrium LTO5 out at 1.5TB RAW/native capacity as their roadmap suggests.
THe uncompressed / RAW data rates of LTO4 Ultrium data tape drives for ARCHIVING material is for me between 105MB-109MB/sec (that's megabytes).. simply BLOWS any firewire 800 interface out of the water. There are many variables however essentially for people like us whose business is content creation or even the part timer who doesn't want to lose stuff, this is really worth a look. Again it will depend on the software you utilise. I use Tolisgroups BRU.
Just do the sums on the time it takes you to archive (or copy to another disk system [BD or DVD-5, DVD 9, et or a spindle) lets say 4 hours of DVCPROHD 1080P (110Mbs or just over 1GB/minute). You can see that this is a lot of content and will take a great deal of time.
This works out at roughly 240GB.
I think this will take over 500 minutes (6 hours? @ 2min/GB)+ to COPY on a FW400 interface and slightly less on a FW800 interface.
However it takes a mere 2200 second or 36 minutes using an LTO4 SAS Ultrium tape drive (with no tape mount/dismount timeloadlocate)!
On a lesser scale, a look at DDS tapes for the low end. These are despite rumours/myths are quite stable and work well with the right software on the mac and the mac laptop.
I am having great success with this archive and recall system using Tolisgroups BRU that is NATIVE for OSX.
I have posted some very recent information on the two SAS PCIe HBAs I am using (ATTOTECH and LSI) and the two tape drives I am using (HP and QUantum).
For LTO4 Ultrium SAS tape drives I would recommend only HP and ATTOTECH EXPRESSSAS as this is reliable and very stable and very fast for OSX.
I am very happy with this archive/recall workflow using LTO4 Ultrium data tape for FCP.
I have post this detail in another forum at
http://www.reduser.net/forum/showthread.php?t=21284&page=6
hth
w
HK

Remove Duplicate Music Files, But Keep iTunes Copy

How can you remove duplicate copies of music files on your hard drive but be sure to keep the one iTunes is using and keeping the iTunes Library intact?
I have hundred of duplicate music files, using a tool like CleanGenius or others I can get a list of duplicate files, but I do not want to break iTunes Library and only want to remove the duplicates that are not being used in iTunes.

I have the same issue. If I have a file in my music folder and then click on it and it opens in itunes it creates a duplicate file in my itunes library. Besides sorting in finder and deleting one by one based upon folder location is there a less mind numbing way to accomplish this task.
Or do I have to pay the $10 for a program like Gemini?

HT2905 No Display Duplicates under File. How to find and remove duplicate items in your iTunes library

I now have iTunes ver 11.0.4.4 under Windows 7. I lost all iTunes stuff when updating to Windows 7. I have loaded thousands amd thousands of music files from backup disks, but there are many duplicates. I am attemping to re-establish my old library. I used to be able to remove duplicates quickly with the old iTunes. The new iTunes doesn't seem to offer the same service. Is there any way to remove duplicates quickly, or must I do it one by one?

When deduping use Shift > View > Show Exact Duplicate Items as this is normally a more useful selection. You need to manually select all but one of each group to remove. Sorting the list by Date Added may make it easier to select the appropriate tracks. If you have multiple entries in iTunes connected to the same file on the hard drive then don't send to the recycle bin. Use my DeDuper script if you're not sure, don't want to do it by hand, or want to preserve ratings, play counts and playlist membership. See this thread for background and please take note of the warning to backup your library before deduping.
(If you don't see the menu bar press ALT to show it temporarily or CTRL+B to keep it displayed)
See also HT2905: How to find and remove duplicate items in your iTunes library
tt2

Removing duplicate files

How can i remove duplicate files. I've al

Do I understand that your music library contains two identical songs and you want to completely get rid of a duplicate so that the file itself won't be stored on the hard drive wherever you keep your actual music library files?
When I delete a duplicate, a second message box comes up and asks if I want to move it to the Trash. Yes isn't the default but Apple +M does it from the keyboard.

Alternate methods of removing duplicate files

Other than what appears in the help (which seems time consuming when you have to go through 9 gigs of songs) is there another way of removing duplicate files?

Do I understand that your music library contains two identical songs and you want to completely get rid of a duplicate so that the file itself won't be stored on the hard drive wherever you keep your actual music library files?
When I delete a duplicate, a second message box comes up and asks if I want to move it to the Trash. Yes isn't the default but Apple +M does it from the keyboard.

How to streamline and remove duplicate files on my HD before I backup?

How to streamline and remove duplicate files on my HD before I backup?
is there a good program or freeware that can automate and isolate when I have
duplicate files hiding in the many folders on my Drive... Yes, in a perfect world
I should know better than having a Drive that is not perfectly organized
I think there is something called SpringCleaning for windows
is there something like this for Mac?
Time machine won't let you remove something after you realized you backed
up two of the same files... Thanks for ideas

+is there a good program or freeware that can automate and isolate when I have
duplicate files hiding in the many folders on my Drive.+
You should back up your drive with Time Machine before you delete any files in case you make a mistake.
Tidy Up! - 1.3.2
http://www.versiontracker.com/dyn/moreinfo/macosx/17908
+Time machine won't let you remove something after you realized you backed
up two of the same files.+
Click on the Time Machine browser icon on the dock. Locate the duplicate file you wish to delete and highlight it. Click on the button with the gear on and choose Delete "all backups of".

Exporting photos for UHDTV or Native 4K TV, what are the best settings ? (File: Quality File: Color Space, Image Sizing and resolution) Or in other words; How can I get the smallest files but keep good quality for display on new UHDTV

Exporting photos for UHDTV or Native 4K TV, what are the best settings ? (File: Quality File: Color Space, Image Sizing and resolution) Or in other words; How can I get the smallest files but keep good quality for display on new UHDTV

You're welcome, and thank you for the reply.
2) Yesterday I made the subclips with the In-Out Points and Command-U, the benefit is that I've seen the clip before naming it. Now I'm using markers, it's benefit is that I can write comment and (the later) clip name at once, the drawback is that I have to view to the next shot's beginning before knowing what the shot contains.
But now I found out that I can reconnect my clips independently to the format I converted the master clip to. I reconnected the media to the original AVI file and it worked, too! The more I work with, the more I'm sold on it... - although it doesn't seem to be able to read and use the date information within the DV AVI.
1) Ok, I tried something similar within FCE. Just worked, but the file size still remains. Which codec settings should I use? Is the export to DV in MOV with a quality of 75% acceptable for both file size and quality? Or would be encoding as H.264 with best quality an option for archiving, knowing that I have to convert it back to DV if I (maybe) wan't to use it for editing later? Or anything else?
Thank's in advance again,
André

Remove duplicates while loading data from text file

Hi,
Data in text file (some times has duplicates) is being loaded into Oracle 9i database using Informatica. To improve performance, we would like to remove duplicates at the time of each load using Oracle procedure. Could you please help me with this?
Thanks,
Lakshmi

No, our table doesn't have that. Most of the functionality is managed at the informatica level. Is there any other way? Thanks,

If I use an app to remove duplicate pix does the app read content or only the file name such as IMG

If I use an app to remove duplicate pix does the app read content or only the file name such as IMG. I have some 2500 pix and the thought of trawling thru them toremove duplicates is mind numbing.
If the apps read the content then I am ok about that but if they only read the file name ... they maybe deleting stuff that isnt really a duplicate

You have to contact with the developer of the application. We do not know which app you are referring to and we do not know how that application works, but all apps of that type are supposed to delete duplicate files with the same format (like IMG, JPG...)

How can I identify and remove duplicate file (audio & photo)

How can I identify and remove duplicate file (audio & photo)? I have quite a few libraries created as backup files.

Terence,
Yes, the duplicates appear in the iPhoto window. I have folders with same name occuring two or even three times sometimes with exactly the same set of photos (ie photos with the same ID) and sometimes with a limited set of photos. Other folders appear only once.
I normally take my MacBook with me if I'm shooting a lot of photos, such as on holiday, and then want to transfer them to my iMac when i get home. Other times I transfer the camera's memory stick directly to my iMac.
I have great difficulty transferring the photos from my MacBook to my iMac. I don't want to store my photos on my iDisk due to the length of time it takes to upload them.
Thanks again.
Simon

Highest quality slideshow saved to movie file is has terrible resolution in PE12, but PDFs look great. What's up?

I've saved a PE12 slideshow as the highest quality available, but the resolution is low when viewing. When saved as a PDF the resolution is outstanding. Does anyone know how I can get better resolution from the movie file?
Message was edited by: Robert Neitz

How are you creating the video file? PSE itself can only record in the horrid outdated VCD format. Output your slideshow to a WMV file and then use PRE, if you have that, or whatever burning software you have to finalize the video.

Remove duplicates, keeping highest quality file.

Similar Messages

Maybe you are looking for