Dealing with duplicates

Hello guys
I am trying to populate a tables with millions of rows, some of the rows have duplicates in the table, i want to try and compare the rows with each other and where a duplicate is found it should skip those rows and not enter them into the table and it's should go to the next and vice versa
Any hints will be appreciated
Thank you

This is the forum for SQL Developer (Not for general SQL/PLSQL questions). Try asking your question in the SQL and PL/SQL forum.

Similar Messages

How is the best way to deal with duplicate photos

I am using a new retina 27" iMac 16gb ram OS X 10.10.1
Aperture 3.6
What is the best way to deal with duplicates that get in Aperture Vaults
I have used Gemini and it finds duplicates, but I have no way of telling if the original are still there.
I don't want to go through 15000 photos to try to find the duplicate.
Thanks Charlie

You mean - one image in a vault, one in a library? Or duplicates in the same library?
Photo Sweeper can scan several libraries or folders at the same time and display the duplicates side by side to let you pick which to keep. You can define rules to mark photos for automatic deletion as well.
http://overmacs.com/photosweeper.html

Tip for dealing with duplicates

Ok we all know it's a big pain in the *** when you get duplicate songs and need to get rid of the duplicates. Especially when you have a LARGE collection like I do.
iTunes clearly has some big shortcoming when detecting and working with duplicates. A lot of the time it'll find stuff that isn't actually a duplicate.
I have some songs that are named the same even though they're not the same song. (Remixes or legitimate duplicates from a "best of..." album or something.)
So I can't just "show duplicates" and delete every other entry indiscriminately. I need a way to go through the list at my leisure (possibly in multiple sittings over several days.) and review all the stuff before I delete it.
Here's a method that makes things easier.
1. Make sure that all your songs have a star rating of at least 1 star. (I default all new songs to my library as 3 stars when importing to itunes and then I raise or lower the stars from my ipod when I listen to them.) If you haven't kept track of your song ratings yet. Give them all a 3 star rating now. The star rating will be used later to select your files to delete.
2. Select your music folder and then select "View/Show Duplicates"
3. From the resulting list, select all songs (Ctrl-A) and then click "File/New Playlist from Selection" Name your new playlist "Suspect dupes" or something
4. Configure your playlist view like this..
* Cover flow on and "View Artwork" on (Selected Item). (You don't want to delete the song with the cover and keep a version with no cover.)
* Make sure you have the following columns visible: Sample rate, Bit rate, Time, Size, Disc #, Track #, Artist, Album, and My Rating. You'll need these to compare the songs.
We now have a playlist which we can refer back to at any time.
Now... If you were to delete files here in this playlist view, all you'd be doing is deleting them from the playlist, and not from your library which is what we want to do.
So...
5. Go back to your library view (Making sure that you've clicked the "Show All" button at the bottom of the window so you're not just seeing the duplicates any more.)
6. Select all the songs in your library (Ctrl-A) and then right mouse button click on the songs and select "Uncheck selection" Which will uncheck every song you have.
7. Go back to your "Suspect dupes" playlist. Everything will be unchecked now. Anything you check while viewing this "Suspect Dupes" playlist will also be checked when you go back to viewing your main library.
8. Here's the tedious part. Go through your "Suspect Dupes" playlist and double click and comparing each potential duplicate. When you find a genuine duplicate that you want to get rid of, Click the box to check the song. Your selections will stay checked even if you close itunes and come back to it another day! (I've been working on my collection for a week now.)
9. When you've finished checking all the ones which are songs you want to get rid of, Go back to the top of the "Suspect Dupes" playlist and scroll through it while Ctrl-Clicking each checked song that's there to highlight the ones you want to get rid of.
10. Once they're all highlighted select "File/New Playlist From Selection" and name the resulting playlist "Delete me" or something. Highlight all the songs in the "Delete me" playlist and then right mouse button click on the songs and select "My Rating/None"
(We do this so that we have a way to recognize these songs when viewing the main library list which is where we can actually physically delete the files to the Recycle Bin from.)
11. Go back to your library view and sort your list by ratings. Highlight all the songs there with a rating of None and then right click to delete them.
12. They're gone, and you're done!
13. As a preventative measure you could Consolidate your library (Advanced/Consolidate Library) and then give all your MP3's a tag in the "Comments" field of something unique like say... "Permanent_File" That way if iTunes should somehow re-add files from somewhere else on your computer again and create more duplicates, you have a way of easily sorting and distinguishing which are the good files and which are ones that you need to delete. (Anything without your "Permanent_File" tag.)
Apple really needs to refine the duplicate detection and make it easier to get rid of them.
* They should make it so that iTunes checks ALL tag info in addition to the song's title when compiling it's list of duplicates.
* There should be a "Select all checked files" option. (So as to avoid the whole star rating thing mentioned above.)
* There should also be an option to remove the files from your library when deleting songs from a playlist view. (eg: A prompt "Delete from this Playlist or Delete from your Library/iPod?"
Custom PC Windows XP Pro Got myself a refund on Vista. Getting a Mac!
Custom PC Windows XP Pro Got myself a refund on Vista. Getting a Mac!

My pet peeve is that when you have a studio recording
and a live recording of the same song by the same
artist, iTunes considers them to be duplicates.
Yeah, that's exactly what I'm saying.
The best way to bring your suggestions to Apple's attention is by submitting the form on this Support page .
I took your suggestion and reffered them to this page. Maybe they'll listen. (Heck I managed to convince the Google Earth team to move the on screen controls to the upper right corner from the bottom center to accomodate dual monitor users.)
Custom PC Windows XP Pro Got myself a refund on Vista. Getting a Mac!

ITunes - Two Libraries by Mistake on same iMac, How to Amalgamate and Deal with Duplicates?

I'm using iTunes 9.2.1 on a PPC iMac G5 running OS X 10.4.11.
I have previously had lots of problems with a damaged iTunes library: https://discussions.apple.com/thread/2009146?threadID=2009146&stqc=true
As a result of this (and I'm not sure how), all music I have added since I got it working again has been stored in a folder called "iTunes Music" (Macintosh HD:Users:Username:Desktop:iTunes Music) which appeared on the Desktop. I also have various iTunes Library files (plus .itdb and .xml files) on the Desktop (I didn't put them there, that's just what appeared).
All music added priror to this is in Macintosh HD:Users:Username:Music:iTunes:iTunes Music.
Everything works in iTunes (as you would expect - it just points to the relevant folder).
I would like to amalgamate the two iTuens Music folders and file all the music in the correct place (wherever that may be) - is there a simple way of doing this?
Another problem I have is duplicated music - I imported into iTunes lots (probably 20+ Gb) of duplicate albums at a higher bitrate than I previously had and deleted the existing lower-bitrate version from iTunes. Unfortunately, I didn't realise that, because of the two iTunes music folders, the older version is still stored in the older folder, even though iTunes isn't using it.
Is there a simple way to identify and remove the older lower-bitrate music from the machine without searching every single album in Spotlight?
I've spent an awfully long time correcting and completing track names, information and artwork in iTunes so I really don't want to lose any of that in the process.
Thanks in advance ...

I totally feel your pain. I too have a similar setup, a G4 running Tiger wherein I ripped my vinyl to a higher bitrate than my CD copies. I have in one of thses threads a script that was written for me that would rummage across the home network and pull down all the lower bitrate copies and move them, but I had a hard drive snafu, ran out of space and never got around to it..
I DO think however, that you can display the bitrate of the files in the list view options within iTunes and sort the files that way, and move the duplicate files you don't want to the trash. From within iTunes.. Let me go fire up the livingroom frankenbox to check..
Yup.. If you use the View Options you can sort through the master list in iTunes by bitrate..
http://imageupper.com/i/?S0300010100011D1344782818223713
Takes a bit to sort out the tracks but it's workable..
As to joining the libraries that you've gotten them cleaned up, you need to drop all the song files into a single folder and let iTunes sort them into the default location. There is a submenu here:
http://imageupper.com/i/?S0500010010011A1344783379227569
http://imageupper.com/i/?S0500010010021A1344783379227569
Good luck!
Deb.

HT204655 How does photos deal with duplicate photos

I Have multiple iPhoto libraries and an Aperture library. How are duplicates photos handled when I upgrade to Photos?

There will be no duplicates unless they are in a Library already. Upgrading to Photos does not merge multiple libraries. If you want to do that, do so prior to upgrading.

How to deal with duplicate packages in AUR?

What is the best way to report a duplicate package (often one version is orphaned and another has a maintainer... but both are still the same package) so that one (if one is orphaned, probably the orphaned one) can be deleted? Would I want to contact a TU? And how would I do that? Post in the Discussions section?

liquibyte wrote:Actually I would say that if you know they are the "exact" same thing, just adopt the orphan and then delete it.
Only TU's can delete packages now. A safety measure to stop the evil minority...

CS 5.5 Design Standard installed. Dealing with duplicate software...

So Now I have Bridge CS5 and CS5.1, Device central CS5 and CS5.5 and on and on and on. What is the solution for this? Part of that comes from also having CS5 Web Premium installed earlier on the machine but one would expect that the installer would notice the outdated version and update them rather than having two sets for everything.

Adobe apps always install full new versions. I uninstalled any CS5 apps that weren't updated to CS5.5 before installing CS5.5. I'm not too sure what implications there are in uninstalling those 5.0 apps now. It's been know to cause some damage to the newer applications so I'd just leave them be.
Bob

Dealing with 6,000 duplicates

Is there any way to get ride of 6,000 duplicate songs on an external harddrive. The Itunes duplicate button shows over 12,000, so do I have to delete every other song right? If i delete all in the window I understand i will lose the original and the duplicate. In otherword can i tell it to just show duplicates, then i can erase all? This is taking me forever hours over hours.

Here is another option that I just figured out while trying to deal with my 15,000 duplicates. The file naming structure of all my duplicates ended in either "song title 1.mp3, song title 1.m4p or song title 2.mp3, song title 2.m4p, etc." depending on how many duplicates there were of each file and the audio format. I used spotlight to search my music folder by name, using the additional constraint of "ends with" and searched for all files that "end with" "1.mp3, 1.m4p, etc."
This identified all of my duplicates which I then dragged into the trash. After removing the files from the music folder, I then went back into iTunes and ran the "Super Remove Dead Tracks" script from the Doug's Apple Scripts website:
http://dougscripts.com/itunes/scripts/ss.php?sp=removedeadsuper
This script removed all of the now dead files that iTunes could not find, due to their removal from my music folder/library.
That solved the problem without having to do the manual selection of tracks in iTunes or my music folder.

My computer died and I had to buy another one. Now I can't sync my ipod classic. What is the deal with this? I added songs about three months ago and it was fine. Now it won't sync. It says it has a duplicate file.

My computer died and I had to buy another one. Now I can't sync my ipod classic. What is the deal with this? I added songs about three months ago and it was fine. Now it won't sync. It says it can't sync because there is a duplicate file.

There are server problems right now with iMessage effecting some users. See http://www.apple.com/support/icloud/systemstatus/.

I'm trying to transfer music from iTunes on an old PC that uses an external storage device to a new PC (Windows 7) that will use that same external storage device. I am also dealing with new iTunes 11. How do I do this?? What folder does iTunes use?

I'm trying to transfer music from iTunes on an old PC (Windows Vista Home Basic) that uses an external storage device to store the files to a new PC (Windows 7 Starter) that will use that same external storage device. I am also dealing with the new iTunes 11. How can I accomplish this successfully? What folder does iTunes use to store the data in? I've tried several things. Home Sharing caused duplicates but not all songs or apps transfered. It is a large library! I've tried just setting the path in the Advanced Tab of iTunes preferences of the new computer with the external drive connected the same as the path when the external drive is connected to the old computer. This was the best solution so far but still a few artists missing and some apps. Any suggestions?

Here are typical layouts for the iTunes folders:
With iTunes 11 you might also have a Home Videos folder inside iTunes Media.
In the simplest cases you copy the entire iTunes folder from <User's Music> on the source computer to <User's Music> on the target machine, install iTunes, and it "just works"TM.
If the media folder (inside the red outline) has been split out to a separate location then you can copy the library folder (outside the red outline) as before and connect the drive holding the media so that it has exactly the same path as before. If the drive appears as D: on one system and E: on the other then the library won't be able to find the media.
The crucial file is iTunes Library.itl - this contains a record of the tracks that have been added to the library, ratings, play counts, playlists etc.
See also: Make a split library portable.
tt2

What to do with duplicates on my computer?

I'm copying lots of folders of images from one computer to another and lots of these folders have duplicate images inside them, so when I import the new folders those duplicates do not come in. This keeps my Lightroom Catalog nice and neat, however, I now have all of these duplicates taking up space on my computer. Just curious how you deal with this? Do you just leave them alone and pretend that they are not there or do you remove them?
Thanks.

I would do this:
1. Get everything into the catalog except for the duplicates.
2. Duplicate the catalog and previews - this will be the main working catalog that you'll come back to in a mo..
3. Turn off 'don't import suspected duplicates' and import them all in one go. That should populate the Previous Import collection with loads of photos that you know are duplicates.
4. Select all of those duplicates and drag them to a folder called 'Duplicates'.
5. Back that folder up to a DVD or something just in case you missed something, then delete it from the hard drive.
6. Delete the temporary catalog you've just been using and go back to your previous one.

How do i deal with multiple iPhoto libraries when migrating to Photos

how do i deal with multiple iPhoto libraries when migrating to Photos

I would merge them before the migration. Also, spend some time doing any batch changes of names and dates beforehand too.
iPhoto Library Manager has good merge and duplicate search facilities.
http://www.fatcatsoftware.com/iplm/

Working with Duplicates

Guys,
We have employee ID as a unique identifer for an employee dimesion.
We have a requirement to report Employee wise data by names which have repetations as data is across Geo's.
I tried sloving this using Alias for employee id with employee name, but it does not allow duplicate Alias names.
Any other way of dealing this to report Employee name?
How to work with duplicates? Is it recommended?

most people append the employee ID to the end of the name alias so every entry is unique.I am giving some detailed picture of Glenn's comments.
It is regular issue when we are handling Customer and Employee kind of dimensions.
Say you are having two columns in the dimension file like
EMPID EMPNAME and EMPNAME had duplicates.
One standard technique is there that we have used in many cases to eliminate the duplication.
While developing rule file you can create another field by joining both the EMPID, EMPNAME fileds.
This new fileld is concatination of both the EMPID, EMPNAME fileds and each attribute in this field will become unique as EMPID is unique.
Do the Mapping as below.
EMPID - GEN2,EMP
EMPNAME - Ignore
NewField - ALIAS2,EMP

MY system folder, including hidden folders are littered with duplicate folders and files with the suffix (from old Mac) How do I remove them

I have a Macbook Pro running Leopard 10.5.8. I had a problem with my my operating system (my fault, I moved a file I shoudnt have) couldnt boot up but was able to boot up from a backup. I managed to repair my original system except now all the system folders, including hidden folders are littered with duplicate folders and files with the suffix (from old Mac). For the most part the dupes are an exact copy, but not always. I want to remove them to free up space and cant imagine duplicate folders in the /system/library are not hindering my computer. But I dont know where to start and am afraid of doing irreparable damage. Any ideas

pacull,
Use iCal>View>Show Notifications to choose what to do with the notification.

Literally the worst customer support experience I've ever had to deal with -- Why don't they let their customers speak to supervisors on demand?

Call 1: I spoke to them to see if I was eligible for an upgrade. It turned out they had accidentally counted a temporary phone switch to an old device as an upgrade. The representative was extremely slow minded and took over an hour to read through my notes and process what I was saying happened, consulting with his supervisor multiple times. He claimed to have noted everything and told me I had to go to a verizon store to verify in person that I was still using my original phone, and that then I would be able to get an upgrade immediately.
Trip to the store: Drove an hour only to be told by the store rep that there were NO NOTES on my account about the issue and there was nothing he could do to help me and that I'd have to call customer support again. The store was closing in 20 mins and customer support seems to take 45 minutes minimum to resolve anything, so I left.
Call 2: Asked a woman to speak to a supervisor about the guy who wasted 2+ hours of my life and gas money. She put me on hold for a long time, came back and said they were all busy right now, would I like to leave my complaint with her? I said no thank you, I will keep waiting. She said ok, and proceeded to hang up.
Call 3: Asked a different woman to speak to a supervisor, she put me on hold for a long time and then told me that since I wasn't an authorized user I could not speak to one... lol ok. Had a family member call and authorize me.
Call 4: Asked a guy to speak to a manager. He insisted on taking the complaint himself. Why would I trust someone to take a complaint about their colleague possibly sitting a few desks away when many of them have proven to be incredibly incompetent already? It sounded like we were making progress... then he hung up too.
Call 5: Asked yet another woman to speak to the manager. She demanded a reason, I told her it was to make a complaint and she told me about a dozen times that the managers would just tell her to take the complaint. I insisted, she refused... finally I threatened to close all of our accounts unless she let me speak to a manager. She said they couldn't right now since the office was closing soon, but that one would call me within 24-48 hours. I'd bet my life savings that never happens. At that point I tried to talk to her about the original upgrade issue, and she said her system had suddenly crashed and that I'd have to call back the next day... Seriously???
Absolutely unreal how horrible their customer support is. Will be switching to T-Mobile immediately.

Having worked in customer service, I have some experience with this kind of situation. Reps are typically trained to make all attempts to handle complaints by themselves, because there are obviously many more reps than supervisors. They're usually REQUIRED to fully authorize the account and to get the whole situation from the customer prior to referring you to a supervisor. One of the biggest problems is that many customers think that demanding a supervisor is a way to guarantee you will get what you want. If the reason for your request to escalate isn't a valid one, the supervisor won't give in to you either.
There are plenty of times in my own experience, especially as a brand new rep out of training, that I took a long time to research and get an understanding of the issue. Yes, I wasn't moving at light speed, but that doesn't make me an idiot. It means I was trying to gather as much info s possible to try and deal with the issue. But I had plenty of customers tell me I was stupid and to transfer them to my supervisor, simply because I took longer than their patience could tolerate to get all the details I needed.
I have never had a job that I executed flawlessly day in and day out. Maybe you have and that's why you can't tolerate a person taking a little longer to wrap their head around the issue, but I'd rather doubt it.
I also don't think you will find reps who work at light speed and utilize a magic wand to fix people's issues at T-Mobile, either, but I guess it never hurts to try.
Just to clarify: I do not work for Verizon and never have. I worked for a residential phone and internet company.

Dealing with duplicates

Similar Messages

Maybe you are looking for