[solved] Did rsync behaviour change? (= nope, not here at least )

Question

Hi!
I just used one of my "full system backup" hd's and the backup script on it deleted all the backups and itself (still made a current backup, just the incremental ones are gone now) -.-
I didn't use that script from the hd for a while (didn't have changes in big files for almost a year, so I only pushed my home folder to the router hd every day), but back when I was working with big files more, I used it every second day for several years and it always worked reliably.
That being said: My guess would be, that it deleted the "excluded=" folders, which is shouldn't (?) or didn't use to with --delete (the hd root was a clone of my system root, except for one excluded /BACKUP directory that kept the increments from --backup). Can't see any other way.
Did anyone notice a change in rsync behaviour? Anything strange?
Couldn't find anything anything on the net that fits... but I almost remember having fixed something rsync-related somewhere... some time ago... not sure what/if/why though, maybe just déjà-vue...
I have some more distributed rsync scripts and might have to check after a lot of things if there's really been a change in rsync behaviour...
( Please excuse the general fuzzyness of this question/topic... I'm in the middle of one of those weird fits of "I should have made a backup before I started that backup now I need to check EVERYTHING and the server too and omg when was the last time I even ssh'd into the router and looked at the cronjobs!?" )
Last edited by whoops (2015-01-31 09:55:19)

Answer 1

\hbar wrote:I don't get it: I see a hyphen before the options. Was it edited?
yes, sorry,  it probably wasn't clear...:
whoops wrote:Oh, thanks, that was a hasty copy paste error (was scared of the --delete ), fixed it now.
... what exactly I "fixed" there. Should have re-posted the line instead of editing the old post .
\hbar wrote:Anyways, it seems to me that */BACKUP/* would not exclude /BACKUP : the /* files you are sync-ing get expanded into their names, one of which would be 'BACKUP' which does not (and should not) match '*/BACKUP/*'
Hmmm, right, "/BACKUP" should definitely be in there too just to be safe, thanks!
There also shouldn't have been a /BACKUP in the first place, though. Here's a more recent version of the whole script that I found on a different backup drive in the meantime (the one that caused the trouble should be almost identical):
#!/bin/bash
# check if root
[ -d /BACKUP ] && { echo "TRYING TO RUN SCRIPT ON WRONG DRIVE?"; exit 1; }
[ -f /backupthis ] || { echo "NOPE! This drive does not want to be backed up ('/backupthis' missing)"; exit 1; }
# find backup drive marker file
while read mount;
do
if [ -a "$mount/BACKUP/backupdriveisabove" ];
then echo $mount;
backup_drive=$mount;
fi;
done < <(mount | grep /media | sed "s/.*on \(.*\) type.*/\1/")
# check if marker file is fine - should match:
# directory="/media/BACKUPMOUNT"; blkid $(mount | grep "$directory") | sed 's/.*: //' > $directory/BACKUP/backupdriveisabove
grep "$(blkid $(mount | grep $backup_drive) | sed 's/.*: //')" "$backup_drive/BACKUP/backupdriveisabove" \
|| { echo "$backup_drive is not a valid backup drive." ; exit 1; }
echo -e "\E[32m===== Backup drive found in $backup_drive =====" && tput sgr0
# Help! I'm scared!
for countdown in `seq 0 5`; do echo "death and destruction in `expr 5 - $countdown`"; sleep 1; done;
# Rsync stuff of which incremental backups should be kept in ./BACKUP/$(date +%Y-%m-%d) first
echo -e "\E[32m===== Partial incremental backup =====" && tput sgr0
synctime=1337
while (( $synctime > 15 ))
do statime=$(date +%s);
rsync -aAXvxbiH /* \
$backup_drive --backup-dir=$backup_drive/BACKUP/$(date +%Y-%m-%d) \
--log-file="$backup_drive/BACKUP/$(date +%Y-%m-%d).log" \
--exclude-from="/opt/scripts/backup.exclude" \
--delete --fuzzy \
|| { echo 'incremental backup failed' ; exit 1; }
eval synctime=`expr $(date +%s) - $statime`
if (($synctime > 14)); then echo "rsync took too long ($synctime) - repeating"; fi;
done;
# Rsync (almost all) previously excluded files / nonincremental full backup to backup drive root
echo -e "\E[32m===== Full system backup =====" && tput sgr0
rsync aAXv /* "$backup_drive" --exclude={share/Trash/*,/dev/*,/proc/*,/sys/*,/tmp/*,/tmp/.*,/run/*,/mnt/*,/media/*,/lost+found,/home/*/.gvfs,*/BACKUP/*,/home/*/.ccache/*,/home/*/.thumbnails/*,/var/tmp/*,/home/*/.mozilla/firefox/*.default/Cache/*,/home/*/.mozilla/firefox/*.default/thumbnails/*,/home/*/.pulse/*,/home/_*,*.part} \
|| { echo 'full backup failed' ; exit 1; }
# Try to merge two old backup increments without making too big a gap (if there are more than 60 backups)
echo -e "\E[32m===== Trying to merge 2 backup increments =====" && tput sgr0
minnamecur=0;
treshold=$(ls -d $backup_drive/BACKUP/2???-??-?? | wc -l);
if (($treshold > 60))
then {
treshold=`expr $treshold / 3`
# Do not create gaps between incremental backups that would be bigger than this many seconds
mingap=$[60*60*24*30]
pushd "$backup_drive/BACKUP" || { echo 'incremental backup folder not found' ; exit 1; }
c=0
# find backup that would leave the smallest gap
for dir in 2???-??-??;
do c="`expr $c + 1`";
dirsec=$(date -d "$dir 00:00" +%s)
eval name_$c=$dirsec;
if (($c > 4)) && (($mingap > 86399));
then {
cpre="`expr $c - 4`";
ccur="`expr $c - 3`";
cnex="`expr $c - 2`";
eval namecur=\$name_$ccur;
eval namepre=\$name_$cpre;
eval namenex=\$name_$cnex;
gap=`expr $namenex - $namepre`;
if (("$gap" < "$mingap"))
then {
mingap="$gap"
export minnamecur="$namecur"
export minnamenex="$namenex"
daysgap=`expr $mingap / 86400`
} fi;
# reduce acceptable gap size for newer backup increments depending on amount of backups
gapreduce="`expr $mingap / $treshold`";
mingap="`expr $mingap - $gapreduce`";
# echo "DEBUG: $c -- Folder $namecur -- acceptable gap size reduced to: $daysgap days";
} fi;
done;
# merge 2 backup increments / remove the old one.
if (( $minnamecur > 2000 ))
then {
eval rmback=$backup_drive/BACKUP/$(date -d "@$minnamecur" +"%Y-%m-%d");
eval toback=$backup_drive/BACKUP/$(date -d "@$minnamenex" +"%Y-%m-%d");
cp -vlrTf $toback $rmback &&\
rm $toback -rv &&\
mv -v $rmback $toback &&\
echo -e "\E[32m===== moved $rmback to $toback, resulting gap: $daysgap days =====" && tput sgr0
} else echo "Could not find Folder to delete. Maybe gaps between backups are too big already."
fi;
} fi;
popd
# install grub to backup-drive
echo grub-install $(mount | grep $(stat -c%m "$backup_drive") | sed "s/[0-9].*//")
# unmount the backup drive :3
umount $backup_drive -v \
|| { echo "Something might have gone horribly wrong but now it's too late to do anything about it :D" ; exit 1; }
From what I can tell, the only way this can has happened is if somehow /BACKUP got created while the script was already running? Or is there another way that I can't see?
Have to retrace my steps...
1) Wanted to make a backup of my project in kdevelop with the builtin git function. When I pressed the "git button" in kdevelop, it crashed and all source files of the project were deleted.
2) temporarily killed git & other backup solutions. Tried to restore files with git manually, but there were no deleted files registered / all commits were empty for some reason.
3) I connected a SPARE_HD and dumped /dev/sda2 (my swap) to it (used a lot of ram so I hoped the source files were in there)
4) Connected OLD_BACKUP_HD in addition to SPARE_HD and started backup script ("just in case")
5) used extundelete to recover everything deleted withing the last 2 hours to the SPARE_HD
6) Started '# strings /sda | grep -n600 "Text that I luckily had in the first line of all my lost files" > /SPARE_HD/big_text-file'
7) Started rummaging trough the stuff that extundelete recovered on the SPARE_HD to see if anything's missing (only 2 of the deleted files were unrecoverable)
8) Started rummaging trough /SPARE_HD/big_text-file with grep, found the last 2 files I was missing in there (HUZZAH!)
9) Looked at my OLD_BACKUP_HD and saw it had deleted its own incremental BACKUP folder plus itself and its logs...
Hmmm, I can totally not see myself creating /BACKUP there somewhere between step 4 and 9 because during that time I tried to avoid writing ANYTHING to /dev/sda, but I don't see another way this could have happened, so I guess I must have done it anyway -.-
*sigh* Maybe it's time to retire that thing... even though it had a pretty good run (several years on several mostly unsupervised machines including a webserver and a router - and no problems)...
And HOW is it possible that EVERY SINGLE time I loose files it's because something goes wrong with an attempt to make a BACKUP?
The only reason why I even need backups seems to be that I keep deleting files while trying to make backups (even if for a change I don't use any messy self-written scripts) and then I end up doing "good old messy file recovery" instead anyways -.-
Last edited by whoops (2015-01-31 08:32:59)