[solved]bash help please: sort files & recognize duplicates

Question

Hi!
I'm trying to move/rename all files in a folder and its subfolder by age.  And move duplicate files somewhere else. The output looks mostly fine, the tmp files too... but I can't figure out, why the duplicate files handling isn't working. edit: the line that sets the new name (&& newname="$1/.dupes/$dname/$part1_$part2_$fname" \) just doesn't seem to get executed here...
#!/bin/bash
pushd "$1" && {
rafi="/tmp/$RANDOM.size.md5.list" && echo > "$rafi"
find $1 -type f -print0 | while read -d $'\0' file;
do \
fname=$(basename "$file")
dname=$(dirname "$file")
part1=$(stat -c%y "$file" | awk -F " |:|\\\.|-" '{printf $1"-"$2"-"$3}')
part2=$(stat -c%y "$file" | awk -F " |:|\\\.|-" '{printf $4"-"$5"-"$6}')
echo "$bname" | grep -q "$part1" && part1=""
echo "$bname" | grep -q "$part2" && part2=""
newname="$1/$part1/$part2_$fname"
size="size$(stat -c%s "$file")"
grep -q "$size" "$rafi" \
md5="md5$(md5sum -b "$file" | sed "s/ .*//")"
grep -q "$md5" "$rafi" \
&& newname="$1/.dupes/$dname/$part1_$part2_$fname" \
|| {
echo "$md5" >> "$rafi"
} || {
echo "$size" >> "$rafi"
newname=$(echo "$newname" | sed "s/\/\/*/\//g" | sed "s/\.~.~//g")
if [ "$newname" != "$file" ];
then if ! [ -d $(dirname "$newname") ]
then echo mkdir -p $(dirname "$newname") -v
fi
echo mv "$file" "$newname" -v --backup=numbered
fi
done
popd
What am I doing wrong? Been trying for hours - I just don't get it...
Thanks!
Last edited by whoops (2011-03-01 10:19:48)

Answer 1

thx, that brought me a little closer!
juster wrote:I don't usually say this for bash but your style is hard to read.
Oops... thought so - never had an easy time writing readable code. Or readable anything... Maybe this is not as bad:
#!/bin/bash
if pushd "$1"
then
# random tmp file for filesize + md5
rafi="/tmp/$RANDOM.size.md5.list" && echo > "$rafi"
# for all files in folder $1 + subfolders...
find $1 -type f -print0 | while read -d $'\0' file
do
fname=$(basename "$file")
dname=$(dirname "$file")
# Get year-month-day
part1=$(stat -c%y "$file" | awk -F " |:|\\\.|-" '{printf $1"-"$2"-"$3}')
# Get hour-minute-second
part2=$(stat -c%y "$file" | awk -F " |:|\\\.|-" '{printf $4"-"$5"-"$6}')
# Don't add this info to new filename if it is already in old filename
echo "$bname" | grep -q "$part1" && part1=""
echo "$bname" | grep -q "$part2" && part2=""
# Destination filename
newname="$1/$part1/${part2}_$fname"
size="size$(stat -c%s "$file")"
# have files with the same size already been processed?
if grep -q $size "$rafi"
then
md5="md5$(md5sum -b "$file" | sed "s/ .*//")"
# have files with the same md5 already been processed?
if grep -q $md5 "$rafi"
# set different destination for duplicate files
then newname="$1/.dupes/$dname/${part1}_${part2}_$fname"
# write md5 to tmp file
else echo $md5 >> "$rafi"
fi
# write size to tmp file
else echo "$size" >> "$rafi"
fi
# remove double slashes
newname=$(echo "$newname" | sed "s/\/\/*/\//g" | sed "s/\.~.~//g")
if [ "$newname" != "$file" ]
then if ! [ -d $(dirname "$newname") ]
then echo mkdir -p $(dirname "$newname") -v
fi
echo mv "$file" "$newname" -v --backup=numbered
fi
done
popd
fi
Renaming seems to work correctly now, but it's still only giving me one "dupe mv line" for my test in /etc/X11 and it's not giving me any for the many duplicates in my huge digicam folder.
$ sort_by_date /etc/X11/
~
mkdir -p /etc/X11/2011-02-27 -v
mv /etc/X11/xorg.conf.bak /etc/X11/2011-02-27/19-43-36_xorg.conf.bak -v --backup=numbered
mkdir -p /etc/X11/2010-06-21 -v
mv /etc/X11/xorg.conf.d/20-nvidia.conf /etc/X11/2010-06-21/10-12-35_20-nvidia.conf -v --backup=numbered
mkdir -p /etc/X11/2010-08-24 -v
mv /etc/X11/xorg.conf.d/10-evdev.conf /etc/X11/2010-08-24/15-45-35_10-evdev.conf -v --backup=numbered
mkdir -p /etc/X11/2010-09-01 -v
mv /etc/X11/xorg.conf.d/20-nvidia.conf.pacnew /etc/X11/2010-09-01/12-16-10_20-nvidia.conf.pacnew -v --backup=numbered
mkdir -p /etc/X11/2010-11-30 -v
mv /etc/X11/xorg.conf.d/20-keyboard.conf /etc/X11/2010-11-30/10-27-00_20-keyboard.conf -v --backup=numbered
mkdir -p /etc/X11/2010-05-26 -v
mv /etc/X11/xorg.conf.d/10-quirks.conf /etc/X11/2010-05-26/21-51-03_10-quirks.conf -v --backup=numbered
mkdir -p /etc/X11/2010-01-03 -v
mv /etc/X11/xorg.conf /etc/X11/2010-01-03/10-56-25_xorg.conf -v --backup=numbered
mkdir -p /etc/X11/.dupes/etc/X11 -v
mv /etc/X11/xorg.conf.dub /etc/X11/.dupes/etc/X11/2011-02-27_19-43-52_xorg.conf.dub -v --backup=numbered
mkdir -p /etc/X11/2010-07-10 -v
mv /etc/X11/Xsession /etc/X11/2010-07-10/20-01-30_Xsession -v --backup=numbered
mkdir -p /etc/X11/2010-12-22 -v
mv /etc/X11/xinit/xinitrc.d/30-dbus /etc/X11/2010-12-22/15-39-41_30-dbus -v --backup=numbered
mkdir -p /etc/X11/2010-11-28 -v
mv /etc/X11/xinit/xinitrc.d/40-libcanberra-gtk-module /etc/X11/2010-11-28/02-44-08_40-libcanberra-gtk-module -v --backup=numbered
mkdir -p /etc/X11/2010-12-19 -v
mv /etc/X11/xinit/xserverrc /etc/X11/2010-12-19/11-58-47_xserverrc -v --backup=numbered
mkdir -p /etc/X11/2010-11-15 -v
mv /etc/X11/xinit/xinitrc /etc/X11/2010-11-15/11-39-17_xinitrc -v --backup=numbered
~
Still don't get it where this problem could be coming from...
md5sum * 2>/dev/null | sort -k 1 | uniq -d -w 32 | cut -c 35-
That's beautiful, just I have no idea how to use it in the context of what I'm trying to do.
Last edited by whoops (2011-02-28 13:00:27)