How to remove duplicates when serializing?

hi,
I think this has been asked before but have not been able to find the answer in the forums or google..
I'd like to save space when serializing a structure to disk, as the storage medium is slow and it is expensive to deserialize large objects. I know that the thing I'm serializing has the same object duplicated in it several times and that its being written out several times.
is there any way to do this (3rd party libraries etc..)?
ideally you'd also be able to configure whether objects are duplicated or not by identity or .equals equality?
thanks,
asjf

hi,
I think this has been asked before but have not been
n able to find the answer in the forums or google..
I'd like to save space when serializing a structure
to disk, as the storage medium is slow and it is
expensive to deserialize large objects. I know that
the thing I'm serializing has the same object
duplicated in it several times and that its being
written out several times.When you say "same," do you mean by identity or by equals? I'm pretty sure that serialization by default does not duplicate the same object (by identity). If you want to avoid serializing "same by equals" you'll probably have to roll your own, overriding readObject/writeObject or whatever methods the docs say to override for custom serialization.
Be aware though, that "same by equals" doesn't really mean the objects will behave the same way. I'd be very careful using that.

Similar Messages

  • How does remove duplicates option in sort transformation behaves?

    Hi
    I am currently on a performance tuning task and I am in the process of replacing the existing transformation (especially SORT, aggregate since it is blocking) into a Stored Procedures (SP). I am not getting the exact results as that of the SORT transformation (remove duplicates) when using the methods in the SP to remove duplicates such as DISTINCT clause, HAVING COUNT>1 option, ROWNUMBER() using OVER PARTITION BY ORDER BY clauses.
    Could any of one help me how exactly does the SORT transformation with remove duplicate options work in SSIS?
    Please let me know if you need any more information.
    Thanks in Advance
    Mervyn

    Hi
    I am currently on a performance tuning task and I am in the process of replacing the existing transformation (especially SORT, aggregate since it is blocking) into a Stored Procedures (SP). I am not getting the exact results as that of the SORT transformation (remove duplicates) when using the methods in the SP to remove duplicates such as DISTINCT clause, HAVING COUNT>1 option, ROWNUMBER() using OVER PARTITION BY ORDER BY clauses.
    Could any of one help me how exactly does the SORT transformation with remove duplicate options work in SSIS?
    Please let me know if you need any more information.
    Thanks in Advance
    Mervyn
    There's an important thing to understand about using the "remove duplicates" feature of the Sort transform - the results are arbitrary. For example, take the following dataset:
    ID  |   Name
    3    |   Mervyn
    2    |   Bharani
    1    |   Nitesh
    2    |   Jamie
    Now, if you use the Sort transform to sort on ID and then remove duplicates then you *might* end up with the following result:
    ID  |   Name
    1    |   Nitesh
    2    |   Bharani
    3    |   Mervyn
    or, you *might* end up with this:
    ID  |   Name
    1    |   Nitesh
    2    |   Jamie
    3    |   Mervyn
    notice how it arbitrarily picks either "Jamie" or "Bharani" because they both have the same ID. I've never seen any requirement, ever, that says "arbitrarily pick a result, I don't care which one I get".
    I guess the point I'm ultimately trying to make is this - are you *sure* that what you want to do is replicate this behaviour? There is no construct in T-SQL that replicates the arbitrary nature of the Sort transfom (and nor should there be), with T-SQL you absolutely have to tell it whether (in the above example) you would want "Jamie" or "Bharani".
    Hope that helps.
    -Jamie
    http://sqlblog.com/blogs/jamie_thomson/ | http://jamiethomson.spaces.live.com/ | @jamiet

  • How to remove duplicates in iphoto 7.1.5 and aperture 2.1.4 on same hard drive

    How to remove duplicates from iPhoto 7.1.5 and Aperture 2.1.4 on same hard drive?

    For iPhoto duplicate annihalitor is a good solution
    For Aperture it is best to ask in the aperture forum
    LN

  • TS3999 how to remove duplicate Birthday entries in icloud on my PC

    how to remove duplicate Birthday entries in icloud on my PC
    The Birthday entries on my iPad are ok.

    Welcome to the Apple Community.
    The following article(s) may help you.
    Resolving duplicate calendars

  • How to Remove Duplicates Without Changing Album?

    Hello All,
    I see a lot of topics on how to remove duplicate songs but there is one thing I would like to know if its possible...
    I would like to remove my duplicates but I still want to see the songs as part of the albums. Example: lets say the same song is present in 3 different albums. I would like to remove 2 of them but have them still showing up as part of the 3 albums. This way the albums remain "complete".
    Is there any way to tool available that can do that? Maybe keeping something like a symbolic link in place of the songs that were deleted?
    Any help would be appreciated.
    Thanks!

    Not possible and has been asked before.
    MJ

  • How to remove duplicates records from output ?

    how to remove duplicates records from output ?  i used delete adjacent but duplicates records are coming again ..suggest me

    hi shruthi,
    thanks for ur answer ..but duplicates records coming again
    here is my code >> plz check it out
    *& Report  ZCRM_TROUBLE_TICKET
    REPORT  zcrm_trouble_ticket.
    TYPES : BEGIN OF ty_qmih,
            qmnum TYPE qmnum,
            equnr TYPE equnr,
            iloan TYPE iloan,
            ausvn TYPE ausvn,
            ausbs TYPE ausbs,
            auztv TYPE auztv,
            auztb TYPE auztb,
            iwerk TYPE iwerk,
            END OF ty_qmih,
            BEGIN OF ty_qmel,
            qmnum TYPE qmnum,
            qmtxt TYPE qmtxt,
            indtx TYPE indltx,
            priok TYPE priok,
            strmn TYPE strmn,
            strur TYPE strur,
            ltrmn TYPE ltrmn,
            ltrur TYPE ltrur,
            objnr TYPE qmobjnr,
            arbpl TYPE lgwid,
            vkorg TYPE vkorg,
            vtweg TYPE vtweg,
            spart TYPE spart,
            END OF ty_qmel,
            BEGIN OF ty_ihpa,
            parnr TYPE i_parnr,
            parvw TYPE parvw,
            objnr TYPE qmobjnr,
            END OF ty_ihpa,
            BEGIN OF ty_crhd,
            arbpl TYPE arbpl,
            objid TYPE cr_objid,
            END OF ty_crhd,
            BEGIN OF ty_crtx,
            ktext TYPE cr_ktext,
            objid TYPE cr_objid,
            END OF ty_crtx,
            BEGIN OF ty_qmfe,
            fecod TYPE fecod,
            fegrp TYPE fegrp,
            qmnum TYPE qmnum,
            END OF ty_qmfe,
            BEGIN OF ty_qmur,
            urcod TYPE urcod,
            urgrp TYPE urgrp,
            urtxt TYPE urstx,
            qmnum TYPE qmnum,
            END OF ty_qmur,
            BEGIN OF ty_iloa,
            tplnr TYPE tplnr,
            iloan TYPE iloan,
            END OF ty_iloa,
            BEGIN OF ty_output,
            qmnum TYPE qmnum,
            equnr TYPE equnr,
           iloan TYPE iloan,
            ausvn TYPE ausvn,
            ausbs TYPE ausbs,
            auztv TYPE auztv,
            auztb TYPE auztb,
            iwerk TYPE iwerk,
            qmtxt TYPE qmtxt,
            indtx TYPE indltx,
            priok TYPE priok,
            strmn TYPE strmn,
            strur TYPE strur,
            ltrmn TYPE ltrmn,
            ltrur TYPE ltrur,
           objnr TYPE qmobjnr,
           arbpl TYPE lgwid,
            vkorg TYPE vkorg,
            vtweg TYPE vtweg,
            spart TYPE spart,
            parnr TYPE i_parnr,
            parvw TYPE parvw,
            arbpl TYPE arbpl,
           objid TYPE cr_objid,
           arbpl1 TYPE arbpl,
            ktext TYPE cr_ktext,
            fecod TYPE fecod,
            fegrp TYPE fegrp,
            urcod TYPE urcod,
            urgrp TYPE urgrp,
            urtxt TYPE urstx,
            tplnr TYPE tplnr,
            END OF ty_output.
    DATA : it_qmih TYPE STANDARD TABLE OF ty_qmih,
           it_qmel TYPE STANDARD TABLE OF ty_qmel,
           it_ihpa TYPE STANDARD TABLE OF ty_ihpa,
           it_crhd TYPE STANDARD TABLE OF ty_crhd,
           it_crtx TYPE STANDARD TABLE OF ty_crtx,
           it_qmfe TYPE STANDARD TABLE OF ty_qmfe,
           it_qmur TYPE STANDARD TABLE OF ty_qmur,
           it_iloa TYPE STANDARD TABLE OF ty_iloa,
           it_output TYPE STANDARD TABLE OF ty_output,
           wa_qmih TYPE ty_qmih,
           wa_qmel TYPE ty_qmel,
           wa_ihpa TYPE ty_ihpa,
           wa_crhd TYPE ty_crhd,
           wa_crtx TYPE ty_crtx,
           wa_qmfe TYPE ty_qmfe,
           wa_qmur TYPE ty_qmur,
           wa_iloa TYPE ty_iloa,
           wa_output TYPE ty_output.
    INITIALIZATION.
      REFRESH : it_qmih,
                it_qmel,
                it_ihpa,
                it_crhd,
                it_crtx,
                it_qmfe,
                it_qmur,
                it_iloa,
                it_output.
      CLEAR:     wa_qmih,
                 wa_qmel,
                 wa_ihpa,
                 wa_crhd,
                 wa_crtx,
                 wa_qmfe,
                 wa_qmur,
                 wa_iloa,
                 wa_output.
    start-of-selection.
      SELECT  qmnum
              equnr
              iloan
              ausvn
              ausbs
              auztv
              auztb
              iwerk
              FROM qmih
              INTO  TABLE it_qmih.
      SORT it_qmih BY qmnum .
      DELETE ADJACENT DUPLICATES FROM it_qmih COMPARING qmnum equnr iloan ausvn ausbs auztv auztb iwerk.
      SELECT   qmnum
               qmtxt
               indtx
               priok
               strmn
               strur
               ltrmn
               ltrur
               objnr
               arbpl
               vkorg
               vtweg
               spart
               FROM qmel
               INTO TABLE it_qmel
               FOR ALL ENTRIES IN it_qmih
               WHERE qmnum = it_qmih-qmnum.
      SORT it_qmel BY qmnum.
      DELETE ADJACENT DUPLICATES FROM it_qmel COMPARING  qmnum
                                                         qmtxt
                                                         indtx
                                                         strmn
                                                         strur
                                                         ltrmn
                                                         ltrur
                                                         objnr
                                                         arbpl
                                                         vkorg
                                                         vtweg
                                                         spart.
      IF it_qmel IS NOT INITIAL.
        SELECT parnr
               parvw
               objnr
               FROM ihpa
               INTO TABLE it_ihpa
               FOR ALL ENTRIES IN it_qmel
               WHERE objnr = it_qmel-objnr.
      ENDIF.
      DELETE ADJACENT DUPLICATES FROM it_ihpa COMPARING  parnr
                                                         parvw
                                                         objnr.
      IF it_qmel IS NOT INITIAL.
        SELECT arbpl
               objid
               FROM crhd
               INTO TABLE it_crhd
               FOR ALL ENTRIES IN it_qmel
               WHERE objid = it_qmel-arbpl.
      ENDIF.
      DELETE ADJACENT DUPLICATES FROM it_crhd COMPARING  arbpl
                                                         objid.
      IF it_qmel IS NOT INITIAL.
        SELECT ktext
               objid
               FROM crtx
               INTO TABLE it_crtx
               FOR ALL ENTRIES IN it_crhd
               WHERE objid = it_crhd-objid.
      ENDIF.
      DELETE ADJACENT DUPLICATES FROM it_crtx COMPARING ktext
                                                        objid.
      IF it_qmih IS NOT INITIAL.
        SELECT fecod
               fegrp
               qmnum
               FROM qmfe
               INTO TABLE it_qmfe
               FOR ALL ENTRIES IN it_qmih
               WHERE qmnum = it_qmih-qmnum.
      ENDIF.
      SORT it_qmfe BY qmnum.
      DELETE ADJACENT DUPLICATES FROM it_qmfe COMPARING fecod
               fegrp.
      IF it_qmih IS NOT INITIAL.
        SELECT urcod
               urgrp
               urtxt
               qmnum
               FROM qmur
               INTO TABLE it_qmur
               FOR ALL ENTRIES IN it_qmih
               WHERE qmnum = it_qmih-qmnum.
      ENDIF.
      SORT it_qmur BY qmnum.
      DELETE ADJACENT DUPLICATES FROM  it_qmur COMPARING urcod
                                                         urgrp
                                                         urtxt.
      IF it_qmih IS NOT INITIAL.
        SELECT  tplnr
                iloan
                FROM iloa
                INTO TABLE it_iloa
                FOR ALL ENTRIES IN it_qmih
                WHERE iloan = it_qmih-iloan.
      ENDIF.
      DELETE ADJACENT DUPLICATES FROM it_iloa COMPARING tplnr
                                                        iloan.
      LOOP AT it_qmih INTO wa_qmih.
        wa_output-qmnum = wa_qmih-qmnum.
        wa_output-equnr = wa_qmih-equnr.
       wa_output-iloan = wa_qmih-iloan.
        wa_output-ausvn = wa_qmih-ausvn.
        wa_output-ausbs = wa_qmih-ausbs.
        wa_output-auztv = wa_qmih-auztv.
        wa_output-auztb = wa_qmih-auztb.
        wa_output-iwerk = wa_qmih-iwerk.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_qmel INTO wa_qmel WITH KEY qmnum = wa_qmih-qmnum.
        wa_output-qmtxt = wa_qmel-qmtxt.
        wa_output-indtx = wa_qmel-indtx.
        wa_output-priok = wa_qmel-priok.
        wa_output-strmn = wa_qmel-strmn.
        wa_output-strur = wa_qmel-strur.
        wa_output-ltrmn = wa_qmel-ltrmn.
        wa_output-ltrur = wa_qmel-ltrur.
       wa_output-objnr = wa_qmel-objnr.
       wa_output-arbpl = wa_qmel-arbpl.
        wa_output-vkorg = wa_qmel-vkorg.
        wa_output-vtweg = wa_qmel-vtweg.
        wa_output-spart = wa_qmel-spart.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_ihpa INTO wa_ihpa WITH KEY objnr = wa_qmel-objnr.
        wa_output-parnr = wa_ihpa-parnr.
        wa_output-parvw = wa_ihpa-parvw.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_crhd INTO wa_crhd WITH KEY objid = wa_qmel-arbpl.
        wa_output-arbpl = wa_crhd-arbpl.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_crtx INTO wa_crtx WITH KEY objid = wa_crhd-objid.
        wa_output-ktext = wa_crtx-ktext.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_qmfe INTO wa_qmfe WITH KEY qmnum = wa_qmih-qmnum.
        wa_output-fecod = wa_qmfe-fecod.
        wa_output-fegrp = wa_qmfe-fegrp.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_qmur INTO wa_qmur WITH KEY qmnum = wa_qmih-qmnum.
        wa_output-urcod = wa_qmur-urcod.
        wa_output-urgrp = wa_qmur-urgrp.
        wa_output-urtxt = wa_qmur-urtxt.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
        READ TABLE it_iloa INTO wa_iloa WITH KEY iloan = wa_qmih-iloan.
        wa_output-tplnr = wa_iloa-tplnr.
        APPEND wa_output TO it_output.
        CLEAR wa_output.
      ENDLOOP.
      DELETE ADJACENT DUPLICATES FROM it_output  COMPARING        qmnum
                                                                  equnr
                                                                  ausvn
                                                                  ausbs
                                                                  auztv
                                                                  auztb
                                                                  iwerk
                                                                  qmtxt
                                                                  indtx
                                                                  priok
                                                                  strmn
                                                                  strur
                                                                  ltrmn
                                                                  ltrur
                                                                  vkorg
                                                                  vtweg
                                                                  spart
                                                                  parnr
                                                                  parvw
                                                                  arbpl
                                                                  ktext
                                                                  fecod
                                                                  fegrp
                                                                  urcod
                                                                  urgrp
                                                                  urtxt
                                                                  tplnr.
    *CALL FUNCTION 'STATUS_TEXT_EDIT'
    EXPORTING
      CLIENT                  = SY-MANDT
      FLG_USER_STAT           = ' '
       objnr                   =
      ONLY_ACTIVE             = 'X'
       spras                   = en
      BYPASS_BUFFER           = ' '
    IMPORTING
      ANW_STAT_EXISTING       =
      E_STSMA                 =
      LINE                    =
      USER_LINE               =
      STONR                   =
    EXCEPTIONS
      OBJECT_NOT_FOUND        = 1
      OTHERS                  = 2
    *IF sy-subrc <> 0.
    MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
            WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
    *ENDIF.
    *CALL FUNCTION 'READ_TEXT'
    EXPORTING
      CLIENT                        = SY-MANDT
       id                            =
       language                      =
       name                          =
       object                        =
      ARCHIVE_HANDLE                = 0
      LOCAL_CAT                     = ' '
    IMPORTING
      HEADER                        =
    tables
       lines                         =
    EXCEPTIONS
      ID                            = 1
      LANGUAGE                      = 2
      NAME                          = 3
      NOT_FOUND                     = 4
      OBJECT                        = 5
      REFERENCE_CHECK               = 6
      WRONG_ACCESS_TO_ARCHIVE       = 7
      OTHERS                        = 8
    *IF sy-subrc <> 0.
    MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
            WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
    *ENDIF.
    *LOOP AT IT_OUTPUT INTO WA_OUTPUT.
    *WRITE : / WA_OUTPUT-qmnum,
             WA_OUTPUT-equnr,
             WA_OUTPUT-iloan,
             WA_OUTPUT-ausvn,
             WA_OUTPUT-ausbs,
             WA_OUTPUT-auztv,
             WA_OUTPUT-auztb,
             WA_OUTPUT-qmtxt,
             WA_OUTPUT-indtx,
             WA_OUTPUT-strmn,
             WA_OUTPUT-strur,
             WA_OUTPUT-ltrmn,
             WA_OUTPUT-ltrur,
             WA_OUTPUT-objnr,
             WA_OUTPUT-arbpl,
             WA_OUTPUT-parnr,
             WA_OUTPUT-parvw,
             WA_OUTPUT-objid,
             WA_OUTPUT-ktext,
             WA_OUTPUT-fecod,
             WA_OUTPUT-fegrp,
             WA_OUTPUT-urcod,
             WA_OUTPUT-urgrp,
             WA_OUTPUT-urtxt,
             WA_OUTPUT-tplnr.
    *ENDLOOP.
      CALL FUNCTION 'GUI_DOWNLOAD'
        EXPORTING
      BIN_FILESIZE                    =
          filename                        = 'E:\CRM1.TXT'
      FILETYPE                        = 'ASC'
      APPEND                          = ' '
         write_field_separator           = '|'
      HEADER                          = '00'
      TRUNC_TRAILING_BLANKS           = ' '
      WRITE_LF                        = 'X'
      COL_SELECT                      = ' '
      COL_SELECT_MASK                 = ' '
      DAT_MODE                        = ' '
      CONFIRM_OVERWRITE               = ' '
      NO_AUTH_CHECK                   = ' '
      CODEPAGE                        = ' '
      IGNORE_CERR                     = ABAP_TRUE
      REPLACEMENT                     = '#'
      WRITE_BOM                       = ' '
      TRUNC_TRAILING_BLANKS_EOL       = 'X'
      WK1_N_FORMAT                    = ' '
      WK1_N_SIZE                      = ' '
      WK1_T_FORMAT                    = ' '
      WK1_T_SIZE                      = ' '
      WRITE_LF_AFTER_LAST_LINE        = ABAP_TRUE
    IMPORTING
      FILELENGTH                      =
        TABLES
          data_tab                        = it_output
      FIELDNAMES                      =
    EXCEPTIONS
      FILE_WRITE_ERROR                = 1
      NO_BATCH                        = 2
      GUI_REFUSE_FILETRANSFER         = 3
      INVALID_TYPE                    = 4
      NO_AUTHORITY                    = 5
      UNKNOWN_ERROR                   = 6
      HEADER_NOT_ALLOWED              = 7
      SEPARATOR_NOT_ALLOWED           = 8
      FILESIZE_NOT_ALLOWED            = 9
      HEADER_TOO_LONG                 = 10
      DP_ERROR_CREATE                 = 11
      DP_ERROR_SEND                   = 12
      DP_ERROR_WRITE                  = 13
      UNKNOWN_DP_ERROR                = 14
      ACCESS_DENIED                   = 15
      DP_OUT_OF_MEMORY                = 16
      DISK_FULL                       = 17
      DP_TIMEOUT                      = 18
      FILE_NOT_FOUND                  = 19
      DATAPROVIDER_EXCEPTION          = 20
      CONTROL_FLUSH_ERROR             = 21
      OTHERS                          = 22
      IF sy-subrc <> 0.
    MESSAGE ID SY-MSGID TYPE SY-MSGTY NUMBER SY-MSGNO
            WITH SY-MSGV1 SY-MSGV2 SY-MSGV3 SY-MSGV4.
      ENDIF.

  • How to remove duplicates that appear on my Library?

    How to remove duplicates that appear on my library?

    On the PC or Mac in iTunes, in the File-menu, select "Show Duplicates". Clean them there and resync to your iPod.

  • How to remove duplicate songs in iTunes?

    Please advise me how to remove duplicate songs in iTunes? Thank you

    THIS is how you remove them, Once you done this...Press and Hold Command on All Music and then Delete and your Done.

  • HT2513 How 2 remove duplicates in iCalendar? I used mobile me 1x n it massed up d things. I even hv 25 duplcts on my entries n on my iPad as well..

    How 2 remove duplicates in iCalendar? I used mobile me 1x n it massed up d things. I even hv 25 duplcts on my entries n on my iPad as well..

    How 2 remove duplicates in iCalendar? I used mobile me 1x n it massed up d things. I even hv 25 duplcts on my entries n on my iPad as well..

  • How to eliminate duplicates when merging photos?

    How to eliminate duplicates when merging photos?

    OK, time to move over to the Aperture Discussions Group.
    It's at this link - https://discussions.apple.com/community/professional_applications/aperture
    This group you are in now is for Lion discussions, and over in the Aperture group, you'll meet skilled people who will help.  Especially if you state your questions clearly and provide sufficient context such that the reader can figure out what is needed.
    Make sure you spend some time reading the Aperture User Guide before asking really basic questions, and search the PDF version using Preview.
    Also search that group for your question, it may well have been asked and answered

  • HT204150 My contacts and groups are not duplicated in iCloud but my groups are duplicated on iPhone and iPad. Any ideas on how to remove duplicate groups or why they are duplicating?

    My contacts and groups are not duplicated in iCloud but my groups are duplicated on iPhone and iPad. Any ideas on how to remove duplicate groups or why they are duplicating?

    Welcome to the Apple Community.
    Check that you don't have more than one account in your contacts, such as 'iCloud' and 'On My Phone'.

  • How we remove duplicate segments from an idoc

    how we remove duplicate segments from an idoc.

    Hi Manju.
    Go to T-code WE19------->Give your Existing Idoc Number or Basic Type --->Get in Select your (Nodes) segment and Click on "Delete Indicator"
    Regards,
    Seegal

  • HT2905 How to remove duplicate songs from the iPhone when the duplicates are not listed in the iTunes library on the computer.

    How can I remove duplicate songs from the iPhone4. The duplicates are not listed in the computer iTunes library.

    The simplest thing may be to back-up and then restore the iPhone. If you don't fancy that then remove all media that is currently synced to it, then double check and manually remove any media that remains on the device, before reselecting your sync options.
    If you've been manually managing the device and your library doesn't contain copies of all your media then see Recover your iTunes library from your iPod or iOS device.
    tt2

  • How to remove duplicate items ?

    ok so ive moved my iTunes library from the NAS drive I bought (after finding out that wouldnt work) onto my new laCie external drive, but ive found some of my albums have triple copies?
    i know how to show duplicates but im not sure how to safely remove duplicates without deleting all copies?

    Oh boy.
    I'm sure there are better ways to do it than this and it will take time, but to avoid all possible loss of data, what I would do is first consolidate all of your libraries.
    • Open the iTunes Library you think is most correct by holding down the OPTION key when you open the iTunes application, which should bring up a dialogue like this: http://cl.ly/image/2i2Q3o0Z0Y3C
    • Then go to preferences and make sure it looks like this: http://cl.ly/image/2C2Z0u0C3T3c
    I would recommend keeping your main iTunes library on your main hard drive (for me that's my internal), unless you definitely can't fit it.
    • Now go to the File menu >Library > Import Playlist
    • Navigate to another of the libraries, and click on the iTunes Library.xml file and import it. Do this for each of your libraries, except the one you are currently using.
    • Now that you've got everything imported into the one library, the fun part starts.
    Do what i said in my previous post and remove all the duplicates.
    • Once that's done, check all the other libraries to make sure you haven't missed anything, and send them on their way to the Trash, and empty it to reclaim all that space.
    I really hope you get this sorted, I went through an ordeal like this myself recently, so it's going to take time, but it feels good when it's all cleaned up and finished!
    xeni
    PS. After writing all this I thought, hmm, why didn't I just Google it instead of figuring it out myself? ;P
    I found this, and it might help if my instructions weren't clear enough. https://bitly.com/LpqFPq
    Also, if you don't already, I urge you to use Time Machine backup. Read more here: http://www.apple.com/osx/apps/#timemachine and http://pondini.org/TM/FAQ.html

  • How to remove duplicate songs from Itunes

    I've consolidates my music in Itunes 11.01. Playlists are just getting crazy with duplicates.
    I want to eliminate duplicate music files, as well as eliminate the references in the interface.
    Then sync my new IPad.
    When I manually erase a duplicate in Windows Explorer, I'm left with a "!" indication in Itunes that the file cannot be found... duh.
    I want to find either an easier way to eliminate duplicates without Windows Explorer, or
    a way to select all the "!" references in Itunes interface and delete.
    Help please from a completely organized person who hates Apples "intelligence" at thinking it knows what I want, and not providing
    adequate help under "remove duplicates" in its help index.
    Thanks.

    How to find and remove duplicate items in your iTunes library - http://support.apple.com/kb/HT2905
    Posts by turingtest2 about different types of duplicates and techniques - https://discussions.apple.com/thread/3555601 and https://discussions.apple.com/message/16042406 (Note: The DeDuper script is for Windows).
    May 2014 post on iCloud duplicates - https://discussions.apple.com/message/25867873
    Show exact duplicates (Mac and Windows) - https://discussions.apple.com/message/16951281
    http://dougscripts.com/itunes/itinfo/dupin.php (commercial)
    iTunes Duplicate Song Manager - http://sourceforge.net/projects/itunesdsm/
    http://www.hardcoded.net/dupeguru_me/
    http://www.wideanglesoftware.com/tunesweeper/index.php
    http://www.araxis.com/find-duplicate-files (commercial, free trial) - finds duplicate files on computer, not specifically iTunes
    I would do it manually.  As you can see from turingtest2's post, some duplicates are not really duplicates.
    I would also review how you add media to iTunes.  It should not be producing all that many duplicates.

Maybe you are looking for