Can I copy files – but with certain restrictions?

Question

I have just finished a large project, the archives of which involves about 5000 "base" files, stored on about 80 CDs and 50 DVDs, involving about 50,000 files in total. Each of the "base" files may have had up to 30 incremental versions. i.e. a certain text file may have undergone revision 23 times, and each revision was saved and archived to (probably) a different disk, with a different suffix – a, b, c and so on. But sometimes the suffix didn't change even though the file was edited. I might have done a bit more dust removal on an image and just overwrote the old file (already archived), and so the new one was archived on a different disk.
I now have 130 disks from which I would like to extract all the files and collapse them to one large archive that will probably span about 20 disks by the time I delete some files not needed. That way I can easily search for all versions of, say, GB097, by going to the particular DVD that has the "G" files on it. Up would come:
GB097
GB097a
GB097b
GB097b-1
GB097b-2
GB097c
... and so on.
This is what I would like to do:
1. Grab the first archive disk, open every folder, and copy all the files to the one folder on a hard drive.
2. Open the second disk and repeat step (1), but with these two provisos.
(a) If a file is identical to a previously copied file (maybe I archived it twice), the file isn't copied. However...
(b) If a file has the same name as a previously copied file, but the data within that file is different (i.e. I removed some dust from an image file, but left the name unchanged), I'd like that file to be copied with a numbered suffix, the same way that Trash treats identically named files.
Any suggestions how I could do this?

Answer 1

Back again. This is the Terminal output when run on my +Duplicates Copy+ folder:
Last login: Thu May 20 13:28:30 on ttyp1
Welcome to Darwin!
jenny-pearces-imac-g5:~ Jenny$ chmod +x /Users/Jenny/Desktop/Bash\ Test\ Documents/BashTest.txt
jenny-pearces-imac-g5:~ Jenny$ /Users/Jenny/Desktop/Bash\ Test\ Documents/BashTest.txt /Users/Jenny/Desktop/Duplicates\ copy/
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 1: ${filePaths${count}}: bad substitution
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 17: fileSum0=d41d8cd98f00b204e9800998ecf8427e: command not found
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 1: ${filePaths${count}}: bad substitution
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 17: fileSum1=d41d8cd98f00b204e9800998ecf8427e: command not found
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 1: ${filePaths${count}}: bad substitution
/Users/Jenny/Desktop/Bash Test Documents/BashTest.txt: line 17: fileSum2=d41d8cd98f00b204e9800998ecf8427e: command not found
jenny-pearces-imac-g5:~ Jenny$
I'm not sure it's worth spending much more time on this. However, if you're prepared to keep posting, JJJ, I'll keep testing.
*SCRIPT AFTER THE EDIT*
#!/bin/bash
declare -i count=0
declare -a fileSum filePaths
# Main folder can be declared on the command line or if not then use /testpix
MainFolder="${1:-/testpix}"
# Set the internal field separator to newline to preserve spaces in file paths
IFS=$'\n'
# Use 'find' to create a list of files within the folders.
filePaths=( $( find "${MainFolder}" -type f \! -name ".*" ) )
# Get an MD5 checksum for each file's combined content of both data and resource forks
for file in ${filePaths[*]} ; do
fileSum${count}=$( cat "${filePaths${count}}" "${filePaths${count}}/rsrc" | md5 | cut -d'=' -f 2)
let count+=1
done
# For each file, check for a duplicate checksum and if found, move the matching file to the user's trash folder
# Rename files with duplicate names by appending #XX
for ((i=0;i<${count};i++)) ; do
[ -z "${filePaths[${i}]}" ] && continue
dupecount=1
for ((j=0;j<${count};j++)) ; do
[ -z "${filePaths[${j}]}" ] && continue
if [ "${fileSum[${i}]}" = "${fileSum[${j}]}" -a ${i} -ne ${j} ] ; then
mv "${filePaths[${j}]}" ~/.Trash && filePaths[${j}]='' && fileSum[${j}]=''
elif [ $(basename "${filePaths[${i}]}") = $(basename "${filePaths[${j}]}") -a ${i} -ne ${j} ] ; then
let dupecount+=1
dirname=$(dirname "${filePaths[${j}]}")
fullfilename=$(basename "${filePaths[${i}]}")
extension="${fullfilename##*.}"
filename="${fullfilename%.*}"
until [ ! -e "${dirname}/${filename} #${dupecount}.${extension}" ] ; do
let dupecount+=1
done
mv "${filePaths[${j}]}" "${dirname}/${filename} #${dupecount}.${extension}"
filePaths[${j}]="${dirname}/${filename} #${dupecount}.${extension}"
fi
done
done
exit 0