Bulk/Scriptable Image File Deduplication Tools?

*Note: I'm not sure if this should be in Sys. Admin. instead. Please correct me.*
I've got an archive of image files, mostly photos but also vector art, that has gotten thoroughly messed up over years of use by several users. This archive is far too large to handle using something like Digikam which is not, as far as I can tell scriptable. One of the biggest problems is that a single image may have been copied to several different directories, under several different names, possible with different/missing EXIF/IPTC data, possibly in different formats (jpeg, png, Nikon nef, Adobe dng, tiff or svg, eps, pdf).
Digikam has a "fingerprinting" and matching tool which works very well for finding duplicates. This is almost a perfect solution. **However**, I cannot figure out how to use it in a scriptable, headless manner and using it through the Digikam interface would take weeks.
Anyone have a suggestion as to how to deduplicate image files programatically?
Thanks.

lolilolicon wrote:AUR/findimagedupes seems what you have in mind.
justinzane wrote:Sure sounds like it! I'm installing now and will give it a spin. Thanks.
So,
findimagedupes
works well for common image formats, but it cannot handle Nikon (NEF) raw files. After RTFM, the algorithm used is quite clear and simple:
DESCRIPTION
findimagedupes compares a list of files for visual similarity.
To calculate an image fingerprint:
1) Read image.
2) Resample to 160x160 to standardize size.
3) Grayscale by reducing saturation.
4) Blur a lot to get rid of noise.
5) Normalize to spread out intensity as much as possible.
6) Equalize to make image as contrasty as possible.
7) Resample again down to 16x16.
8) Reduce to 1bpp.
9) The fingerprint is this raw image data.
To compare two images for similarity:
1) Take fingerprint pairs and xor them.
2) Compute the percentage of 1 bits in the result.
3) If percentage exceeds threshold, declare files to be similar.
Since I **need** NEF support, and I dislike/suck at Perl, does it seem reasonable to rewrite/wrap the app in shell/Python/C to handle raw files with dcraw/libkdcraw? Or is there an alternative program/library that handles raw files? Suggestions?

Similar Messages

IndesingCS2 server scriptable pluign: how to import image file into a frame.?

Hello I am creating a scriptable pluign for indesingcs2 server. Now I am stuck at importing a image file in a image frame on a document. The code which was running fine for indesingcs2 desktop is given below. /////////////////////////////////////////////////////////////////////////////////////// /////////// IDFile sysFile = SDKUtilities::PMStringToSysFile(const_cast<PMString* >(&ImageFileNamewithcompletepath)); InterfacePtr<ICommand> importCmd(CmdUtils::CreateCommand(kImportAndLoadPlaceGunCmdBoss)); if(!importCmd) return kFalse; InterfacePtr<IImportFileCmdData> importFileCmdData(importCmd, IID_IIMPORTFILECMDDATA); if(!importFileCmdData) return kFalse; //db is input.I got it using the techniques mentioned in the indesign-server-plugin-techniques.pdf //page 18.. importFileCmdData->Set(db, sysFile, kMinimalUI); ErrorCode err = CmdUtils::ProcessCommand(importCmd); if(err != kSuccess) return kFalse; InterfacePtr<IPlaceGun> placeGun(db, db->GetRootUID(), UseDefaultIID()); if(!placeGun) return kFalse; UIDRef placedItem(db, placeGun->GetItemUID()); InterfacePtr<ICommand> replaceCmd(CmdUtils::CreateCommand(kReplaceCmdBoss)); if (replaceCmd == nil) return kFalse; InterfacePtr<IReplaceCmdData>iRepData(replaceCmd, IID_IREPLACECMDDATA); if(!iRepData) return kFalse; iRepData->Set(db, imageBox.GetUID(), placedItem.GetUID(), kFalse); ErrorCode status = CmdUtils::ProcessCommand(replaceCmd); if(status==kFailure) return kFalse; return kTrue; /////////////////////////////////////////////////////////////////////////////////////// ////////// I used the same code for making scriptable plugin for indesignCS2 server as it doesn't involve use of any UI element. But this code is not working.It doesn't crash or returns kFalse. What i see after executing the plugin through script is a grey region for image in the graphics frame. I request if anyone gives me solution to this i will be highly grateful to him. Thanks and Regards, Yogesh Joshi

Actually, your code is working fine, as there are no returned errors, asserts or crashes.
The reason you are just seeing a gray box, is that the display performance is set to the lowest level in InDesign Server for performance reasons.
Hint: Open the document in "desktop" InDesign, right click on the image, and look at the Display Performance settings for the image.
See
Ken Sadahiro, "[CS2 Server] Controlling display performance, how?" #1, 23 Feb 2006 8:01 am
You can of course programmatically change it to the normal or high res settings from within your plug-in, but it might slow down the performance of InDesign Server a bit.
One additional comment:
In your code where you do this:
importFileCmdData->Set(db, sysFile, kMinimalUI);
I would do this, just to be safe:
if (LocaleSetting::GetLocale().IsProductFS(kInDesignServerProductFS )) { importFileCmdData->Set(db, sysFile, kSuppressUI); } else { importFileCmdData->Set(db, sysFile, kMinimalUI); }

Feature request: Better tool for organizing image files

Hi all,
I'm considering rearranging the directory structure of my photo library completely (for a number of reasons: new cameras, a new variety of tools on new PCs).
I would like to reorganize my photos into a year\month directory structure and that will basically spread all my image files into new locations.
I have searched quite a bit and the advices I have seen don't seem to work:
(1) Moving the files outside LR is fairly easy (using exiftool) but I cannot make LR link to the files automatically in their new locations. And doing it manually with "Locate file" or "Find missing folder" is not practical due to the large number of files and large number of new locations.
(2) Moving the files with LR tools is also not practical: Manually dragging the photos into the new directory structure will take forever, and the only automatic rename feature I know of is "Rename Files ..." cannot move the photos into new directories (it can, however, assign names from the file dates and other metadata).
Now my feature suggestion: Either make it possible for LR to automatically relink files that have been scattered into a new directory structure or - better yet - make it possible for LR to perform such auto-organize operations.
My mp3 manager (MediaMonkey) is a good example to follow: It can organize (move or copy) any number of music files into any directory structure built from the audio file's metadata (such as my standard audio directory structure: Music\<Artist>\<Album>\<Track#:2> - <Title>).
Interestingly, the LR "Rename file" function can almost do it already; it just cannot move the file into a new directory.
Or can this be done in a way I haven't discovered?

Sorry if I might be stating the obvious (but at least I was such a *dummy* once that I needed this explicitely):
In order to move the sequence of images per drag&drop in grid view you need to
Put the sort order in the bottom toolbar to "user order" (toggle visibility with letter T if not displayed)
Grab an image from within its thumbnail, not from its border-cell.
When you move around you see a vertical black thick line wandering in between images where you would be able to drop it
Put the images first into a collection in case they should be spread across multiple folders.
This is the reason why most often from a filter result you cannot re-arrange.
Still sometimes it does not work as I want, but can *be persuaded*, i.e. I repeat the same trial after first moving to another collection, then return back. I can't nail it down sufficiently to file a bug.
For an automated sorting according to filename (chosse that setting finally in toolbar) you could rename the photos in place, without exporting & re-importing. Just select them and hit <F2> to bring up the renaming dialog like possible during import.
+1 vote from me, as I am neither satisfied with the responsiveness of moving folders within LR e.g. from laptop hard disk to NAS.
Apart from that I try not to build much on my folder structure, just break them into manageable quantities and have a quick "diary-overview" by using naming convention "YYYY-MM-DD description of event" (pictures of up to 4 cameras go inside, unique filenames for Canons achieved by renaming during import). For all the rest I prefer to build (smart) collections.

Upload Image/File using Sql Developer tool

Is there a way to upload an image/file to a HUGEBLOB field from the sql developer tool?
(Toad has this, we were wondering if this is available on sql developer)
Thanks for your help!
Sunder

K,
Thanks for the answer.
Do you know if you have to restart oracle after loading .jar files ? I ask this because I am not aware if oracle/jvm automatically knows that the jar files have been uploaded.
Thanks

Disc image file exported from Encore CS4 won't work

Hello,
I have a problem concerning the export of a disc image file in Encore CS4. I recently upgraded to CS4 and wanted to continue my project which I started editing with CS3. While working with CS3 I already did image files from this project for burning and testing the DVD. Back then it worked with no problems.
However, since using the CS4 version Encore does the image without any problem, but I can't use the image file in any way afterwards. I mounted it with Daemon Tools as I did before, but the DVD won't play in any software DVD player I have installed (Media Player Classic, Windows Media Player, Nero Showtime) nor can I access the file structure over a file manager. Nevertheless NeroInfoTool and ImgBurn say there is a DVD in the virtual drive. Then I exported the DVD project into a folder and now I can see the files in the file manger but again no player will play it.
Has anyone experienced this before and is able to help? It would be very appreciated since I don't want to downgrade back to CS3 :-)
It's Encore CS4, Version 4.0.1.048 on a 2,8 GHz HT CPU, 2 GB RAM, Nvidia Geforce 6800 UT, lots of HDD space, XP + SP3.
Thank you,
Chris...

Hey,
thank you both. As I said no player accepts the project exported to a folder. It doesn't matter if I point the player to the video_ts.ifo or just the folder - it won't play. As a consequence I re-installed Encore CS3 in which exporting worked fine and guess what - the exported image and folder output doesn't work either. Since it worked before in CS3 it comes down to the conclusion that either I did some editing to the project that screwed the project or Encore CS4 did. Checking for mistakes in the project and previewing in inside CS4 everything seems fine. So might it be CS4 that ****** up the project file??? I mean it's full of strange bugs so it could easily be that. I even updated my Daemon Tools yesterday and it won't make a difference.
Or could it be something else I haven't considered yet? Any ideas? I just don't want to build the whole project again which might not even work if it's some other error. This is a desperate call for help
Thanks,
Chris...

Swapping Macs, can't change the image file used for TM to a Time Capsule

Hello ~
I have ten Macs connected to one TimeCapsule (TC) and nine of them work flawlessly performing a TimeMachine (TM) backup. On the one that doesn't work properly, it is a PBP with Leopard 10.5.8, and latest updates. The other Macs are mixed 10.5.x and 10.6.x.
I had a user on the PBP that I had been backing up in the past. I erased the HD and reinstalled the OS, and then created a new account. I wanted to start backing up the new account and have the name of the backup be the new name I put under Sharing Preference (I assume that's where TM gets the name of the backup).
Well, apparently not, the name of the backup is still the old user name from Sharing, and on the old account before the computer was erased.
I have repaired permissions, I have thrown out the TM prefs in Mac HD\Library\Preferences\, I have restarted the computer. So far I can not get the name to change on the sparse image that it copies to, even though TM says the backup completed without error.
So, as it appears the backup is using the same old image file that this Mac had been using, the backup are separate when you go into TimeMachine, they do not seem to access each other's data, and inspecting the contents of the image, the date of the backup file is there for the new user in the old image file.
I really would just like the PBP to stop using the old image file and create it's own image.
Any ideas welcome!
Thanks all
Chaz

I'm slightly confused. What is a PBP or did you mean a MacBook Pro (MBP)? And, what do you mean by a backup "image?" TM does not make images. Are you actually referring to the backup folder for this machine?
If I understand correctly what you need to do is delete the old backup for this machine so TM can start from scratch with a new backup. To do that open the TM application and select your backup, then from the Finder's tool icon in the toolbar select "Delete." which should delete the entire backup.
Also, see User Tips for Time Machine for help with TM problems. Also you can select Mac Help from the Finder's Help menu and search for "time machine" to locate articles on how to use TM. See also Mac 101- Time Machine.

Efficiency of decoding and displaying image files?

BRIEF SUMMARY
My question is this: can Flash Player download JPG and GIF
files from a server and rapidly open/decode them, ready for
display, efficiently and
entirely 'in memory'?
Would Flex be a good choice of language for developing a RIA
that needs to continually download lots of JPG and GIF images
on-the-fly from the server and render them on the screen? I *don't*
want my application to thrash the hard disc.
BACKGROUND
I am designing a 'rich' web app, and I'm investigating
whether Flex is the right tool for the job.
Although Flash's animation features are an obvious selling
point for implementing my user interface, I also need to do some
server-side rendering. What I want to do is perhaps a little
unorthodox: I will be generating lots of GIF and JPG files on the
fly and these will be streamed to the client (along with other
application data, e.g. in XML format) to update different parts of
the on-screen document. In need this to happen very quickly (in
some cases, creating the effect of animation).
It happens that JPGs and 16-colour GIFs will be, by far, the
most efficient formats for streaming the images, because of the
nature of the application. I could of course send the images in
some proprietary format, geared for my application, but presumably
decoding the images would be slow as I would have to implement this
myself in ActionScript, and so I would be limited by the speed of
Flex 'bytecode'. (I realise Flash is a lot more optimised than it
once was, but I am hoping to see a gain from using image formats
that Flash natively understands!)
Naturally the internet bandwidth should (in principle) be the
bottleneck. However, assuming I can get my image files to the
client on time, want I want to know is:
how efficient is Flash at loading such files?
Bearing in mind that I'm not just displaying the occasional
image -- I will be doing this continuously. Most of the images
won't be huge, but there will be several separate images per
second.
The image files will be a mixture of normal colour JPGs and
4-bit colour GIFs (LZW-compressed). I know that Flash natively
supports these formats, but depending on how Adobe have implemented
their LZW/Huffman decoding and so on, and how much overhead there
is in opening/processing downloaded image files before they are
ready to 'blit' to the screen, I imagine this could be pretty fast
or pretty slow!
If my client only has a modest PC, I don't want the JPG/GIF
decoding alone to be thrashing his CPU (or indeed the disc) before
I've even got started on 'Flashy' vector stuff.
I'm new to Flash, so are there any 'gotchas' I need to know
about?
E.g. Would it be fair to assume Flash Player will do the
decoding of the downloaded image entirely 'in memory' without
trying to do anything clever like caching the file to disc, or
calling any libraries which might slow down the whole process? It
would be no good at all if the images were first written to the
client's hard disc before being ready to display, for example.
Further, if I'm doing something a little out-of-the-ordinary,
and there is no 'guarantee' that images will be loaded quickly,
what I'm doing might be a bad idea if a later version of Flash
Player may (for example) suddenly start doing some disc access in
the process of opening a newly downloaded image. So, while I could
just 'try it and see', what I really need is some assurance that
what I'm doing is sensible and is likely to carry on working in
future.
Finally, I imagine JPG/GIF decoding could be something that
would vary from platform to platform (e.g. for the sake of
argument, Flash Player for Windows could use a highly-optimised
library, but other versions could be very inefficient).
This could be the 'make or break' of my application, so all
advice is welcome! :) Thanks in advance.

You need a servlet/jsf component to render the image in the response.
Look at this: http://www.irian.at/myfaces-sandbox/graphicImageDynamic.jsf

I am unable to drag and drop image files in a JAVA supposrted site. I have all the lastest versions of softwarte here - everything is up to date.

I am no longer able to drag and drop image files int this JAVA supported site. It had always worked flawlessly in the past.
I would open the site, go to Send Files, allow access, fill out the fields and then drag and drop my image files. Now I can only use the ADD (+) button to do so.
I have the latest versions of OS-X Lion and JAVA installed on my computer
http://www.clippingprovider.com/CP_II_EU/Welcome.html

This is the Tech Sheet on the subject:
Photoshop Help | Managing paths
It contains this item under Manage Paths:
When you use a pen or shape tool to create a work path, the new path appears as the work path in the Paths panel. The work path is temporary; you must save it to avoid losing its contents.
Ok, the red you referred to is a Stoke you added. Then QuickMask is not involved.

The specified resource name cannot be found in the image file.

Dear Experts,
I have developed an import too and it imports many of the Business Objects from a temporary data base to SAP B1. It was running fine for more than 6 months now.
Currently when the Tool is importing Credit Notes in to SAP B1, the following error is thrown.
The specified resource name cannot be found in the image file.
And I have no Clue about the error. Any one had any experience regarding the above error???
Thanks in Advance,
Vasu Natari.

Hello VAsu,
Somethimes this unexpected, ununderstandable eroor messages caused by corrupted observer dll.
Do a clean up (delete %TEMP%SM_OBS_DLL) folder, and see the result ! (i had several issues in the past with some strange messages...)
Regards
János

How to update image files in web pages generated by servlet without redeploying

HI,
I have a servlet which generates html pages with images. Those images changes
as they are graph generated regularly. They are placed in s directory where the
application is deployed. It is necessary because I have made some security constraints
for the location of those files. However it seems that when I deploy my application
on WLS all files contained in the application directory are cached, and when for
ex. I delete some of them they are still appear in generated html pages. only
Redeployment of application updates the state of those files.
Is there a way to make the application to be aware of any changes in images files
and to load updated ones??
i would be interested in a situatioin like with default WLS servelt (DefaultWeb
Application) where any new files are seen as soon as they are placed in Default
WebApplication directory.
Is there a way to periodically redeploy application from CLI?
thx in advance
Michal

889096 wrote:
Hello!
In SQL Developer when we run htp package then we see the generated code in OWA Output, But how to invoke or show the webpage by Oracle tools/plsql program.
and
Without copying the code and save it to textfile.html and open it with browser.
I heard it is possible by configuring Apache server.
Please give me brief details, as i am beginner with ORACLE 10g
And
If you are using the htp package, you are presumably developing for an application server? Can the people who set that up not help you with your development environment?
Setup is too complicated to cover here. Check out the Oracle HTTP Server documentation on oracle's documentation pages.
Briefly, you need to install Oracle HTTP Server and then configure a data access desciptor to connect to your database.
How to see the generated code in "QUEST TOAD"? I am not able to see OWA output in TOADYou will have to ask this question in a Toad forum.

How can I insert a link to an image file into numbers

I wish to include access to images in a numbers spreadsheet without actually inserting the images. Doing this where the image is a web page is not difficult but these images are on my local drive. I tried dropping an image file onto Safari ( the image opens as a page in Safari) and then copying the URL ( it has the form: file:///Users/JTC_49/Documents/Pictures/IMG_0248.JPG ) and using it to create a link ( Format>Add a Link>paste the URL into the 'link' field). That all seems to work but when you click on the link you get a message " Couldnt Open Link ". A web page link would open a browser and display the linked page. This does not appear to try to open a browser to pass the URL.
MacBook Pro, OS X Mavericks (10.9.3), Numbers 3.2

Open the parent file. Go to Tools> Advanced Editing> Link tool. Then draw an area in the parent file with the link tool to open the new file. You will get the Create link dialog box. Under Link action, select "Open a file" and click on next. Select the file to open and click on Select button. You will get the "Specify Open Preference Window". Select "New Window" and Click on OK. Save the file and re-open it. This time the new file will open in a new window keeping the parent file.
Bobby Johnson.

Possible to scan a document into Acrobat XPro,then replace part of that PDF with text or image file?

Hi everyone,
Is it possible first to scan a paper document into Acrobat X Pro, and then to replace a "pane" or box within that PDF with a text or image file?
Would it require SDK and Javascript?
Thanks,
Rick Weinhaus

CtDave and others:
Here's the problem in more detail.
I am a physician and I already have the ability to incorporate paper-based
documents into an electronic health record (EHR) by scanning them and
saving them as PDFs. The documents are paper patient encounter forms with
a different box or 'pane' for each category of data. Each box has a fixed
location on the paper form.
There are three situations, however, where I want to have the option to
overwrite certain boxes of the PDF with data generated by methods other
than scanning.
1) Insert text macros with variable fields using the electronic health
record's software.
2) Insert a drawing available as a pre-existing JPG or generated by using a
tablet.
3) Insert text generated using Dragon Naturally Speaking.
For efficient office workflow, in all three cases the text or image to be
inserted into the box or 'pane' in the PDF needs to be generated while I am
still seeing the patient -- that is -- before the paper form has been
scanned and converted to a PDF.
The sticky note tool is one option, but I would want to be able to modify
the default size of the sticky note so that it corresponded exactly to the
size of the box it was 'overwriting.' I can't figure out how to do this?
Could it be done using scripting?
Furthermore, I would want the sticky notes to automatically display when
the PDF was opened, so that a hover would close rather than open them. This
sticky note method would have the advantage of allowing the user to toggle
back and forth between inserted data and any original handwritten text.
Any thoughts?
Rick

Opening an image file in an existing TextEdit window

I'm blocked on this, which seems it should be straightforward (but maybe it isn't).
If I have an open TextEdit window on the desktop and drag an image file to it, the image appears in the window.
How do I replicate that result with AS?

Hi red_menace
Your proposed approach seems reasonable, although I haven't tried it. Rather, I'm looking at other ways of achieving the present goal. That is to open a small window on the desktop with an embedded image and embedded text -- and then, at certain dates (not a problem), move the window to a different position, perhaps with a different image and different text.
A TextEdit window certainly wasn't ideal for this, as it has baggage I can do without -- a ruler (which can be eliminated with a system events command), a title, and the three manipulation buttons.
A window that would better meet my needs would be a "Stickies" window, but Stickies isn't scriptable.
I can move it around fine with Finder commands, and I can close it. If I create all the windows I will need ahead of time (there are a limited number), I don't know whether it's possible to bring them up selectively -- I don't even know where/how the files are stored; I've looked at the package contents and don't find them there. I'm talking about the files for Stickies window that appear when Stickies is launched.
The only other app I've found with a fairly basic window layout is Core Image Fun House.app, which also isn't scriptable -- but maybe I can get a bead on its stored files.
Or maybe there are other choices . . . ?
I'd welcome suggestions but in the meantime will just keep playing around with it. This is not the most urgent matter in the world!
osimp

Can't move image files within Bridge

I recently bought an iMac and got the CS5 license transferred over to the Mac platform. Got CS5 loaded and everything works except for one critical thing: I can no longer move image files within Bridge. When I do the right click on an image file to select that command, the command is "grey" and won't let me to choose it. So I am forced to rearrange my image files through the Mac's Finder/Pictures feature. Drag and drop doesn't work either. I have looked everywhere for a setting that would allow movement of files to different folders and rearrangement within Bridge, and can't find one. I have Bridge set as my default for photos and not the lame iPhoto program on Macs. I previously was to be able to freely use Bridge for organizing my photos on a PC and now sorely disappointed that I can't do this with my new iMac.

I previously was to be able to freely use Bridge for organizing my photos on a PC and now sorely disappointed that I can't do this with my new iMac.
No need for disappointment because you can move all files freely within Bridge.
First check if you have read and write permissions for your files/folders/disk (try move the files using finder) If Finder allows you to drag the files to the location you previously wanted in Bridge then the problem lies in Bridge.
First select a file and with right mouse click menu choose purge cache for selection. If the problem is gone you could do so for the whole folder using menu tools/cache/purge cache for folder while having Bridge pointed to that folder.
If not restart bridge holding down option key and choose reset preferences.
could you report back?

Can I back up individual image files to a slideshow DVD created in Premiere Elements 4.0?

I created a Premiere Elements 4.0 Project including 3 separate photo slideshows. The slideshows were created in Photoshop Elements 6.0 and imported into my Premiere Elements project. I was hoping that when burning the project to DVD, my image files would also be individually backed up.
When I give the DVD to family members I would like them to have access to the image files so they can print or use them individually. I would like to do this in 1 single DVD rather than giving them a DVD with the slideshows and a DVD with the image files. I was able to do this with a previous DVD authoring program I was using, but I cannot figure it out with Premiere Elements 4.0.
Any ideas if this is do-able with Premiere Elements 4? (I am running Windows Vista Home premium SP1 - 64bit - 3GB RAM)

Thanks for your help. I tried this. I burned the project to a folder (4.7GB) then to that folder I added a folder with my photos. So in the folder there was "Photos", "OpenDVD" and "Video_TS". I had 3 separate folders within the "Photos" folder, to keep the 3 slideshow photos grouped.
I burned the entire folder to a DVD using my burn tool. I don't even know the name of the software I used, but I think it was just the Toshiba burn tool. I dragged and dropped the folder into my new disc folder, then at the top of my screen clicked "burn to disc", I named the disc and chose from the formating options "Mastered: disc can be read on other computers and some CD/DVD... no new data can be added..."
Once burned I checked it on my DVD player and now it recognizes only the individual jpg files so it does a generic slideshow of these images rather than my slideshow project. (no menu, music...) Putting the disc in my computer I could access either the files or the slideshow project, but not on my home DVD player.
I burned the folders to 2 separate discs, once dragging the whole folder to the disc and once opening the folder and dragging only the individual folders to the disc.
Please tell me what I'm doing wrong. thanks.

Bulk/Scriptable Image File Deduplication Tools?

Similar Messages

Maybe you are looking for