Backpac: A package state snapshot and restore tool for Arch Linux

Question

backpac:
A package state snapshot and restore tool for Arch Linux with config file save/restore support.
https://aur.archlinux.org/packages.php?ID=52957
https://github.com/altercation/backpac (see readme on the github repository for more information)
Summary & Features
It's a common method of setting up a single system: take some notes about what packages you've installed, what files you've modified.
Backpac creates those notes for you and helps back up important configuration files. Specifically, backpac does the following:
maintains a list of installed groups (based on 80% of group packages being installed)
maintains a list of packages (including official and aur packages, listed separately)
maintains a list of files (manually created)
backs up key config files as detailed in the files list you create
The package, group and files lists along with the snapshot config files allows system state to be easily committed to version control such as git.
Backpac can also use these lists to install packages and files. Essentially, then, backpac takes a snapshot of your system and can recreate that state from the files and lists it archives.
Use Cases
Ongoing system state backup to github
Quick install of new system from existing backpac config
Conform current system to given state in backpac config
Backpac is a very, very lightweight way of saving and restoring system state.
It's not intended for rolling out and maintaining multiple similar systems, it's designed to assist individual users in the maintainance of their own Arch Linux box.
Status
Alpha, release for testing among those interested. Passing all tests right now but will continue to rework and refine. Bug reports needed.
Why?
There are a lot of 'big-iron' solutions to maintaining, backing up and restoring system state. Setting these up for a single system or a handful of personal systems has always seemed like overkill.
There are also some existing pacman list making utilities around, but most of them seem to list either all packages or don't separate the official and aur packages the way I wanted. Some detect group install state, some don't. I wanted all these features in backpac.
Finally, whatever tool I use, I'd like it to be simple (c.f. the Arch Way). Lists that are produced should be human readable, human maintainable and not different from what I'm using in non-automated form. Backpac fulfills these requirements.
Regarding files, I wanted to be able to backup arbitrary system files to a git repository. Tools like etckeeper are interesting but non /etc files in that case aren't backed up (without some link trickery) and there isn't any automatic integration with pacman, so there is no current advantage to using a tool like that. I also like making an explicit list of files to snapshot.
Sample Output
This is the command line report. Additionally, backpac saves this information to the backpac groups, packages and files lists and the files snapshot directory.
$ backpac -Qf
backpac
(-b) Backups ON; Files will be saved in place with backup suffix.
-f Force mode ON; No prompts presented (CAUTION).
(-F) Full Force mode OFF; Prompt displayed before script runs.
(-g) Suppress group check OFF; Groups will be checked for currency.
(-h) Display option and usage summary.
(-p) Default backpac: /home/es/.config/backpac/tau.
-Q Simple Query ON; Report shown; no changes made to system.
(-R) Auto-Remove OFF; Remove/Uninstall action default to NO.
(-S) System update OFF; No system files will be updated.
(-U) backpac config update OFF; backpac files will not be updated.
Sourcing from backpac config directory: /home/es/.config/backpac/tau
Initializing.................Done
GROUPS
============================================================================
/home/es/.config/backpac/tau/groups
GROUPS UP TO DATE: group listed in backpac and >80% local install:
base base-devel xfce4 xorg xorg-apps xorg-drivers xorg-fonts
GROUP PACKAGES; MISSING?: group member packages not installed:
(base: nano)
(xfce4: thunar xfdesktop)
PACKAGES
============================================================================
/home/es/.config/backpac/tau/packages
PACKAGES UP TO DATE: packages listed in backpac also installed on system:
acpi acpid acpitool aif alsa-utils augeas cowsay cpufrequtils curl dialog
firefox gamin git ifplugd iw mesa mesa-demos mutt netcfg openssh rfkill
rsync rxvt-unicode sudo terminus-font vim wpa_actiond wpa_supplicant_gui
xmobar xorg-server-utils xorg-twm xorg-utils xorg-xclock xorg-xinit xterm
yacpi yajl youtube-dl zsh
AUR UP TO DATE: aur packages listed in backpac also installed on system:
flashplugin-beta freetype2-git-infinality git-annex haskell-json
package-query-git packer wpa_auto xmonad-contrib-darcs xmonad-darcs
AUR NOT IN backpac: installed aur packages not listed in backpac config:
yaourt-git
FILES
============================================================================
/home/es/.config/backpac/tau/files
MATCHES ON SYSTEM/CONFIG:
/boot/grub/menu.lst
/etc/acpi/handler.sh
/etc/rc.conf
/etc/rc.local

Answer 1

firecat53 wrote:I think your plan for handling an AUR_HELPER is good. If AUR_HELPER is defined by the user, then either you might need a list of major AUR helpers and their command line switches so you can pick the correct switch for what needs to be done (most use some variation of -S for installing, but not all), or have the user define the correct switch(es) somehow for their chosen AUR helper.
That's a good idea. I'll add that to my AUR refactoring todo.
I also found directory tracking to be a weakness in other dotfile managers that I tried. I think you would definitely have to recursively list out the contents of a tracked directory and deal with each file individually. Wildcard support would be nice...I just haven't personally found a use case for it yet.
I've been thinking that I could just add the directory and scan through it for any non-default attribute files. If those are found then they get automatically added to the files list. That's pretty close to what etckeeper does.
Edit: I just compiled the dev version and removed my comments for already fixed things...sorry!
The master branch should have those fixes as well, but I didn't update the version number in the package build. I'll have to do that.
1. Still apparently didn't handle the escaped space for this item: (the file does exist on my system)
Ok, good to know. This wildcard directory business will require some new code and refactoring so I'll also rework my filenames handling.
2. Suggestion: you should make that awesome README into a man page!
I was working on one (the pkgbuild has a commented out line for the man page) but I had to leave it for later. Definitely want a man page. Once this stabilizes and I'm sure there aren't any big structural changes, I'll convert it to man format.
3. Suggestion: add the word 'dotfile' into your description somewhere on this page, the github page, and in the package description so people looking for dotfile managers will find it. You could also consider modularizing the script into a dotfile manager and the package manager, so people on other distros could take advantage of your dotfile management scheme.
I actually have a different script for dotfile management that doesn't touch packages, but there is definitely overlap with this one. That script isn't released yet, though, and if people find this useful for dotfile management that's great. I'll add that in.
4. Suggestion: since -Q is a read-only operation, why not just make it run with -f automatically to avoid the prompt?
Originally, running backpac without any command line options produced the Query output. I was concerned that since it is a utility that can potentially overwrite system files, it is important to give users a clear statement prior to execution about what will be done. Since the Query output is essentially the same as the Update and System reports in format and content, I wanted to be explicit about the Query being a passive no-change operation. The current command line options aren't set in stone though. If you feel strongly about it being different, let me know.
Long answer to a short question
5. Another suggestion: any thought to providing some sort of 'scrub' function to remove private information from the stored files if desired? This would be cool for publishing public dotfiles to github. Perhaps a credentials file (I did this with python for my own configs). Probably detecting email addresses and passwords without a scrub file would be rather difficult because dotfiles come in so many flavors.
Yes, absolutely. In fact, if you look at the lib/local file (pretty sure it's in both master and dev branches in this state) you'll see some references to a sanitize function. The idea there is that the user will list out bash associative arrays like this:
SANITIZE_WPA_=(
[FILE]='/etc/wpa_supplicant.conf'
[CMD]='sed s/expungepattern/sanitizedoutput/g'
Question: am I missing an obvious option to remove a file from the files.d directory if I delete it from the files list? Or do I have to delete it manually? It might be helpful to add a section to the README on how to update and delete dotfiles from being tracked, and also a more detailed description of what the -b option does (and what is actually created when it's not used).
You are only missing the function I didn't finish. There should be either dummy code or a TODO in the backpac main script referencing garbage collection, which isn't difficult but I just haven't finished it. The idea being another loop of "hey I found these old files in your files.d, mind if I delete them?" It's on my list and I'll try to get it in asap.
And finally, just out of curiosity, why did you choose to actually copy the files instead of symlink like so many other dotfile managers do?
git not following symlinks, hardlinks also out for permissions issues (git wouldn't be able to read the files, change them, etc.)
I definitely would prefer to not make an entire copy of the file, but I haven't come up with a better option. Shout with ideas, though. Also, if there is a way around the link issues I noted above, let me know. I don't see one but that doesn't mean it's not there.
edit: I think a Seattle area Arch meetup would be cool! Perhaps coffee someplace? Bellevue? U-district? Anyone else? BYOPOL (bring your own pimped out laptop)
A general meetup sounds good. I was also thinking it would be fun to do a mini archcon with some demos.