Arch Linux Pacman Proxy Script

Question

Since my Arch box has no internet connection (yes, I'm that much of a masochist), I wrote a bash script to act as a proxy of sorts for pacman, at least as regards downloading. It took a while to get it working (intermittent web access and all) but here it finally is, as promised.
It works well for me, and I only hope that it will help anyone else in the same situation (if there is anyone else in the same situation).
The script has various options to explain everything you need to know, but here's a quick word of warning: it only works on POSIX-compatible computers (i.e. no Wintendows, usually) and the proxy box needs to have bash, wget, tar, grep and sed installed. I don't see this as being a problem, but if I'm wrong, tell me all about it.
#!/bin/bash
readonly VER="2.1"
readonly wgetVER=`wget -V|sed '1!d'`
# DEBUG levels:
# Set to 0 for ALERT-level messages only
# Set to 1 for INFO-level messages (I prefer this)
# Set to 2 for DEBUG-level messages
# Set to 3 for all messages
readonly DEBUG=0
# Set DEBUGLOG=1 to print ALERT, INFO and DEBUG messages to DEBUG.log
readonly DEBUGLOG=0
readonly COLOUR=true
if [[ $COLOUR == true ]] ; then
readonly red='\e[31m'
readonly warn='\e[31;7m '
readonly green='\e[32m'
readonly lgreen='\e[32;1m'
readonly yellow='\e[33;1m'
readonly lblue='\e[34;1m'
readonly dull='\e[0m' ; fi
# Sticky details:
# tar seems very slow (vfat? transform? USB?)
function bugspray {
[[ -z $2 ]] && verbosity=1 || verbosity=$2
if (( DEBUG >= verbosity )) ; then
(( $2 == 0 )) && intro="${warn}ALERT${dull} | "
(( $2 == 1 )) && intro=" INFO | "
(( $2 > 1 )) && intro=" ${yellow}DEBUG${dull} | "
echo -e "$intro$1"
(( DEBUGLOG == 1 )) && echo "`date +%F_%T` | $1">>log-alpps/DEBUG.log ; fi ; }
function compare_versions {
# Requires two ordinary, untweaked version strings, first local, second distant
if [[ -z $2 ]] ; then
bugspray "compare_versions didn't receive two variables." 2 ; fi
update='no'
[[ -z $2 ]] && return
local localversion=(`echo $1|sed -r 's/([[:alpha:]]+)/ \1 /g;s/[\.:~_-]/ /g'`)
local distantversion=(`echo $2|sed -r 's/([[:alpha:]]+)/ \1 /g;s/[\.:~_-]/ /g'`)
local max=$((${#distantversion[*]}))
bugspray "localversion: ${localversion[*]}; distantversion: ${distantversion[*]}" 2
for (( x=0 ; (( x < $max )) ; $((x++)) )) ; do
bugspray "local version part: `echo ${localversion[$x]}` | distant version part: `echo ${distantversion[$x]}`" 2
if [[ `echo ${localversion[$x]}|grep -E '^[0-9]*$'` && `echo ${distantversion[$x]}|grep -E '^[0-9]*$'` ]] ; then
bugspray "Numerical comparison" 2
bugspray "localver (${localversion[$x]}) less than distantver (${distantversion[$x]}): $(( 10#${localversion[$x]} < 10#${distantversion[$x]} ))" 3
# Different compare methods for numbers, single letters and multiple letters.
if (( 10#${localversion[$x]} < 10#${distantversion[$x]} )) ; then
update='yes'
break ; fi ; fi
if [[ `echo ${localversion[$x]}|grep -E '^[[:alpha:]]$'` && `echo ${distantversion[$x]}|grep -E '^[[:alpha:]]$'` ]] ; then # single letter -> compare
bugspray "Lexicographical comparison" 2
bugspray "localver (${localversion[$x]}) less than distantver (${distantversion[$x]}): [[ ${localversion[$x]} < ${distantversion[$x]} ]]" 3
if [[ ${localversion[$x]} < ${distantversion[$x]} ]] ; then
update='yes'
break ; fi ; fi
if [[ `echo ${localversion[$x]}|grep -E '^[[:alpha:]]+$'` && `echo ${distantversion[$x]}|grep -E '^[[:alpha:]]+$'` ]] ; then # string -> drop
# cvs / git / beta -> what are the rules?
bugspray "Sequence of letters. Not treating as version number." 2 ; fi
done ; }
function find_dependencies {
# Determine existence via unique path
bugspray "+ ${lgreen}Building download list for package \"$1\"${dull}" 1
local packagename
local dlfile
local package_found=no
if [[ `ls .temp-alpps/*.db/$1-* 2>/dev/null` ]] ; then
bugspray "Analogous package-name file(s) found" 2
for x in `ls .temp-alpps/*.db/$1-*/desc` ; do
bugspray "Checking $x" 2
packagename=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $x`
if [[ j$packagename == j$1 ]] ; then
bugspray "Exact match found: $packagename = $1" 2
dlfile=`sed '/%FILENAME%/,+1!d;/%FILENAME%/d' $x`
local distantversion=`sed '/%VERSION%/,+1!d;/%VERSION%/d' $x`
local package_full=$packagename-$distantversion
local the_repo=`echo $x|sed -r 's/.temp-alpps\/(.*)\.db.*/\1/'`
local the_path=`echo $x|sed -r 's/desc//'`
unplus=`echo $packagename|sed 's/\+/\\\+/g'`
local local_version=`sed -r '/^'$unplus' /!d;s/.* //' .temp-alpps/snapshot.state`
package_found=yes
break ; fi ; done ; fi
if [[ -z $dlfile ]] ; then
bugspray "Exact match not found -> looking for replacements." 2
if ! [[ -z `grep $1 .temp-alpps/*.db/*/depends` ]] ; then
local provisional=`grep $1 .temp-alpps/*.db/*/depends|sed -r 's/\/depends.*//'`
for package in $provisional ; do
local providence=`sed -r '/%PROVIDES%/,/^$/!d;/%PROVIDES%/d;/^$/d;s/>.*//;s/=.*//' $package/depends`
for y in $providence ; do
if [[ j$y == j$1 ]] ; then
# But what about when several packages provide the same thing and ALPPS picks the wrong one? Can it happen?
local newdep=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $package/desc`
bugspray " -> Package \"$newdep\" provides \"$1\"." 1
find_dependencies $newdep
break 2 ; fi ; done ; done
if [[ -z $newdep ]] ; then
bugspray "${warn}Warning:${dull} package \"$1\" not found in database." 0 ; fi ; fi ; fi
bugspray "package: $1 | package_found = $package_found" 2
# Add file to download_list if not up to date and if not already present
if [[ j$package_found == jyes ]] ; then
local already_got_one=no
unplus=`echo $dlfile|sed 's/\+/\\\+/g'`
[[ `echo $download_files|grep $unplus` ]] && already_got_one="yes" && bugspray "${green}Package in queue${dull}" 1
[[ `ls downloads/$dlfile 2>/dev/null` ]] && already_got_one="yes" && bugspray "${green}Package proxied${dull}" 1
compare_versions $local_version $distantversion
bugspray "compare_versions says: $update" 2
! [[ -z $local_version || $update == yes ]] && already_got_one="yes" && bugspray "${green}Package already installed and up-to-date${dull}" 1
if [[ j$already_got_one != jyes ]] ; then
bugspray "${green}Confirm downloading${dull}" 1
download_files+="$dlfile "
download_array[${dlfile}]=$the_repo
bugspray "Checking for further dependencies" 2
if [[ -a $the_path/depends && `grep -E '^%DEPENDS%$' $the_path/depends` ]] ; then
local dependency=`sed -r '/%DEPENDS%/d;/^$/q' $the_path/depends`
bugspray "Dependencies of \"$packagename\":$dependency" 3
for x in $dependency ; do
# Is this dependency already installed? Sort-of the same as higher up; it just saves time here.
local depname=`echo $x|sed -r 's/>.*//;s/=.*//'`
local depminver=`echo $x|sed -r 's/.*>//'`
bugspray "Dependency name: \"$depname\"" 2
if ! [[ `grep -E '^$depname ' .temp-alpps/snapshot.state` ]] ; then
find_dependencies $depname ; fi ; done
else
bugspray "${lblue}End of the line${dull}: package \"$packagename\" has no dependencies." 2
true ; fi ; fi ; fi ; }
function failover_fetch {
echo "Downloading `echo $1|sed 's/.*\///'`"
[[ -d log-alpps/ ]] || mkdir log-alpps/
[[ -z $success ]] || unset success
for mirror in $(<.temp-alpps/snapshot.mirrorlist) ; do
# If DEBUG>0, shouldn't redirect output
url=`echo $mirror|sed 's/$arch/'$arch'/;s/$repo/'${download_array[$1]}'/'`/$1
bugspray "Connecting to $url" 2
wget -o .temp-alpps/very-temp-log -U "Arch Linux Pacman Proxy Script version $VER / $wgetVER" -P downloads/ $url && success=true
cat .temp-alpps/very-temp-log>>log-alpps/download.log
rm -f .temp-alpps/very-temp-log
if [[ -n $success ]] ; then
bugspray "Download: \$success = true" 2
break ; fi ; done
if [[ -z $success ]] ; then
bugspray "${warn}FAIL:${dull} $1 not accessible on known mirrors." 0 ; fi ; }
case $1 in
warranty)
echo
echo -e " ${green}Warranty${dull}"
echo -e " ${green}========${dull}"
echo " This program is free software. It comes without any warranty, to"
echo " the extent permitted by applicable law. You can redistribute it"
echo " and/or modify it under the terms of the Do What The Fuck You Want"
echo " To Public License, Version 2, as published by Sam Hocevar. See"
echo " http://sam.zoy.org/wtfpl/COPYING for more details."
echo
howto)
echo
echo -e " ${green}How to use ALPPS${dull}"
echo -e " ${green}================${dull}"
echo -e " ${yellow}+ Step 0${dull}: prime the engine"
echo " On your offline box, copy the ALPPS script onto a removable medium, cd into"
echo " it's directory, then run it with the \"init\" option. This essentially"
echo " takes a snapshot of your system, package-wise. Your removable medium is"
echo " now ready for use!"
echo -e " ${yellow}+ Step 1${dull}:"
echo " On the proxy box the first order of business is to download and decompress"
echo " the current package lists. You can do this with the \"fetchdb\" option."
echo -e " ${yellow}+ Step 1 alt${dull}:"
echo " If you want to download the package lists without decompressing them,"
echo " \"fetchdb simple\" will do this. It's a lot quicker, but please keep"
echo " in mind that you cannot then do anything in step 2."
echo -e " ${yellow}+ Step 2${dull}:"
echo " If you want to download packages (and why wouldn't you?), the easiest way"
echo " is to prepare a simple text file containing the names of the packages you"
echo " want, one on each line. Run ALPPS with the option \"fetch <filename>\" and"
echo " sit back and wait until it's done. Don't worry about dependencies: ALPPS"
echo " handles them automatically."
echo -e " ${yellow}+ Step 2 alt${dull}:"
echo " As a convenience, \"fetch full\" will download all the packages needed to"
echo " update the offline box."
echo -e " ${yellow}+ Step 3${dull}:"
echo " Finally, back on your offline box, run ALPPS again with the \"install\""
echo " option. This will update the package list and copy the package files into"
echo " local cache. You can now run \"pacman -S <package names>\" to finish"
echo " installing the packages."
# ALPPS no longer installs the packages, just caches them locally.
# Not entirely sure why; it just felt too klutzy.
echo -e " ${yellow}+ Step 4${dull}:"
echo " You will now probably want to delete all the stuff you no longer need. Run"
echo " ALPPS with the option \"clean\". This won't touch your request file(s). It"
echo -e " ${red}WILL${dull} delete log files, so if you want to keep them, back them up first."
echo " Next time you do this, don't forget to run ALPPS with \"init\" again."
echo
bugs)
echo
echo -e " ${green}Where this goes wrong${dull}"
echo -e " ${green}=====================${dull}"
echo " + ALPPS is a bit slow when packages have many dependencies. It's a recursive"
echo " shell script: what did you expect?"
echo " + It only uses the settings in /etc/pacman.conf. If your conf file is"
echo " elsewhere, you're SOL. Similarly, the repositories are all taken from"
echo " /etc/pacman.d/mirrorlist. If you added any custom repositories in"
echo " /etc/pacman.conf (or any other file), ALPPS ignores them."
echo " + As it stands, ALPPS only works when the proxy box has bash, wget, sed,"
echo " grep and tar installed. The offline box needs bash, pacman and sed (in"
echo " theory, this shouldn't be a problem...)."
echo " + There is as yet no way of handling package groups, short of listing every"
echo " member of the group."
echo " + I don't think ALPPS will ever be able to handle AUR packages. Of course, if"
echo " you're compiling AUR packages, you probably won't need something like this!"
echo
faq)
echo
echo -e " ${green}Frequently Asked Questions${dull}"
echo -e " ${green}==========================${dull}"
echo
echo -e " ${red}Q:${dull} Can I run the whole thing off a USB stick?"
echo -e " ${lblue}A:${dull} Certainly."
echo
echo -e " ${red}Q:${dull} Can I use a non-POSIX computer as a proxy (e.g. Windows)?"
echo -e " ${lblue}A:${dull} Probably not, unless that computer has a POSIX-compatibility layer"
echo " installed (such as Cygwin) with bash, wget, tar, sed and grep."
echo
echo -e " ${red}Q:${dull} Do I need to download the package list, waste time updating my computer,"
echo " then go back again to download the packages I want?"
echo -e " ${lblue}A:${dull} Nope! You can update the packagelist database and download piping-hot"
echo " fresh packages, all in one sitting."
echo -e " ${red}Q:${dull} You mean I won't have to futz around with package lists that keep updating"
echo " ten minutes after I download them?"
echo -e " ${lblue}A:${dull} Exactly. Nice, isn't it?"
echo
echo -e " ${red}Q:${dull} Does this thing handle SSL and signed packages?"
echo -e " ${lblue}A:${dull} ALPPS uses SSL if:"
echo " - the mirror has an https address (at present, none do), and"
echo " - wget on the proxy box is compiled with SSL support"
echo " Concerning signed packages: no, but then again, it doesn't need to. ALPPS"
echo " downloads packages; it's pacman's responsibility to verify them. Your"
echo " system remains safe (or as safe as it ever was, at any rate)."
echo -e " ${red}Q:${dull} What about gpg keys? I need to get them."
echo -e " ${lblue}A:${dull} ...maybe later, say, in version 3."
echo
echo -e " ${red}Q:${dull} ALPPS says it \`prepares databases´. Can pacman still use them after this?"
echo -e " ${lblue}A:${dull} Yes, it can. In fact, what ALPPS does is extract the databases into a"
echo " temp directory, without altering the original .db files."
echo
echo -e " ${red}Q:${dull} Where can I contact you?"
echo -e " ${lblue}A:${dull} For constructive, useful questions and comments: [email protected] and"
echo " be sure to mention Arch Linux in the subject."
echo " For flames, trolling, spam and the like, visit your local bitbucket."
echo
todo)
echo
echo -e " ${green}What's next?${dull}"
echo -e " ${green}============${dull}"
echo -e " + I ${red}might${dull} be able to extend the reach of this thing to the AUR."
echo " Yes, I know I said ALPPS couldn't (in \"bugs\"), but I've learned"
echo " something new since then. It'll be tricky, though, since the AUR web"
echo " interface delivers results in python. While this should be a Good Thing,"
echo " bash isn't very good at handling python-format lists and dicts."
echo
-v|ver|version|--ver|--version)
echo -e "${lgreen}Arch Linux pacman proxy script${dull} | ${yellow}version $VER${dull}"
dl_list) # Debugging
(( DEBUG == 0 )) && echo "the \"dl_list\" option is only for debugging" && exit 0
arch=$(<.temp-alpps/snapshot.architecture)
unset download_files
unset download_array
declare -A download_array
find_dependencies $2
bugspray "Download list: $download_files" 3
echo "Download list v2:"
for x in $download_files ; do
bugspray " File \"$x\" from repo \"${download_array[$x]}\"" 3
echo " Full URL: http://mirror.archlinux.org/${download_array[$x]}/os/$arch/$x" ; done
verint) # Debugging
(( DEBUG == 0 )) && echo "the \"verint\" option is only for debugging" && exit 0
for x in `ls .temp-alpps/*/$2-*/desc` ; do
packagename=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $x`
if [[ j$packagename == j$2 ]] ; then
bugspray "Unique package-name file found: $packagename=$2" 2
distantversion=`sed '/%VERSION%/,+1!d;/%VERSION%/d' $x`
dlfile=`sed '/%FILENAME%/,+1!d;/%FILENAME%/d' $x`
package_full=$packagename-$distantversion
the_repo=`echo $x|sed -r 's/.temp-alpps\/(.*)\.db.*/\1/'`
the_path=`echo $x|sed -r 's/desc//'`
local_version=`sed -r '/^'$packagename' /!d;s/.* //' .temp-alpps/snapshot.state`
break ; fi ; done
echo "full package filename: $dlfile"
echo "Comparing local ($local_version) and distant ($distantversion) versions of $2..."
compare_versions $local_version $distantversion
#compare_versions 2.5beta3-2 2.5cvs4-1
echo "...and the verdict is: $update"
init)
x=`uname -m`
if [[ -x /usr/bin/pacman ]] ; then
[[ -d .temp-alpps ]] || mkdir .temp-alpps/
pacman -Q>.temp-alpps/snapshot.state
sed -r '/^Se/!d;s/Server = //' /etc/pacman.d/mirrorlist>.temp-alpps/snapshot.mirrorlist
sed -r '/^\[/!d;/options/d;s/\[(.*)\]/\1/' /etc/pacman.conf>.temp-alpps/snapshot.repositories
sed -r '/^Architecture/!d;s/^.*= //' /etc/pacman.conf>.temp-alpps/snapshot.architecture
[[ ! -s .temp-alpps/snapshot.architecture || auto == $(<.temp-alpps/snapshot.architecture) ]] && $x>.temp-alpps/snapshot.architecture
sed -r '/^SyncFirst/!d;s/.*= //;s/ /\n/g' /etc/pacman.conf>.temp-alpps/snapshot.prioritypackages
echo "Current state recorded."
else
echo "You seem to be running ALPPs on an unsupported system."
echo "ALPPS is the ${lgreen}Arch Linux Pacman Proxy Script${dull} and simply"
echo "won't run properly on a non-pacman OS (except when proxying)."
echo "Aborting. Sorry."
exit 1 ; fi
fetchdb)
[[ -d downloads ]] || mkdir downloads
arch=$(<.temp-alpps/snapshot.architecture)
declare -A download_array
for repo in $(<.temp-alpps/snapshot.repositories) ; do
download_array[${repo}.db]=$repo
location="${repo}.db"
failover_fetch $location ; done
if [[ j$2 != jsimple ]] ; then
[[ -d log-alpps ]] || mkdir log-alpps
[[ -a syncfirst ]] && rm -f syncfirst
echo "Preparing databases for local processing. This might take a while."
for x in $(<.temp-alpps/snapshot.repositories) ; do
echo "Preparing $x.db"
if [[ -a downloads/$x.db ]] ; then
[[ -d .temp-alpps/$x.db ]] || mkdir .temp-alpps/$x.db/
tar -xzC .temp-alpps/$x.db/ -f downloads/$x.db --transform 's/:/§/' --no-same-owner && echo "$x database ready for use."
else
echo -e "Database $x.db is missing. Skipping. This will probably cause problems." ; fi ; done
# SyncFirst package warning
[[ -z $download_list ]] || unset download_list
[[ -a syncfirst ]] && rm -f syncfirst
for x in $(<.temp-alpps/snapshot.prioritypackages) ; do
find_dependencies $x ; done
bugspray "download_files: $download_files" 2
if [[ ! -z $download_files ]] ; then
echo "New version(s) of SyncFirst package(s):"
for x in $(<.temp-alpps/snapshot.prioritypackages) ; do
[[ ! -z `echo $download_files|grep $x` ]] && echo "- $x" && echo $x>>syncfirst ; done
echo "Package names placed in 'syncfirst'. Don't forget to download them!" ; fi ; fi
fetch)
for x in $(<.temp-alpps/snapshot.repositories) ; do
if ! [[ -d .temp-alpps/$x.db/ ]] ; then
bugspray "You must first download a fresh database with the \"fetchdb\" option" 0
exit 1 ; fi ; done
arch=$(<.temp-alpps/snapshot.architecture)
unset download_array
declare -A download_array
unset download_files
if [[ $2 == full ]] ; then
echo " Building full upgrade list. This will almost certainly take a while."
echo " Please wait..."
for y in `sed -r 's/^(.*) .*/\1/' .temp-alpps/snapshot.state` ; do
bugspray "Examining $y" 1
find_dependencies $y ; done
elif [[ -a $2 ]] ; then
echo " Determining dependencies. This might take a while. Please wait."
for y in $(<$2) ; do
bugspray "Examining $y" 1
find_dependencies $y ; done
else
echo " Error: you should provide a list-file or the keyword \"full\"."
echo " (see 'alpps.sh howto', step 2)"
exit 1 ; fi
if [[ -z $download_files ]] ; then
echo " No files to download: either none found, or all found are up to date."
else
[[ -d downloads ]] || mkdir downloads
for x in $download_files ; do
failover_fetch $x ; done ; fi
install)
dbpath=`sed -r '/^DBPath/!d;s/^.*= //' /etc/pacman.conf`
[[ -z $dbpath ]] && dbpath="/var/lib/pacman/"
bugspray "dbpath: $dbpath" 2
echo " Updating databases"
cp -ft ${dbpath}sync/ downloads/*.db
cachedir=`sed -r '/^CacheDir/!d;s/^.*= //' /etc/pacman.conf`
[[ -z $cachedir ]] && cachedir="/var/cache/pacman/pkg/"
echo " Caching packages"
cp -f downloads/*.pkg.tar.xz $cachedir
echo
echo " Don't forget to install/update your packages."
echo " Exercise proper caution."
echo " Have fun!"
clean)
echo "Cleaning up. This might take a while."
rm -fr .temp-alpps/ && echo "Buffer directory deleted" || echo "Problem deleting buffer directory '.temp-alpps/'"
rm -fr log-alpps/ && echo "Log directory deleted" || echo "Problem deleting 'log-alpps/'"
rm -fr downloads/ && echo "Downloaded packages deleted" || echo "Problem deleting 'downloads/'"
echo
echo -e " ${lgreen}Arch Linux pacman proxy script${dull}"
echo
echo -e " ${green}Description${dull}"
echo -e " ${green}===========${dull}"
echo -e " This is a bash script to fetch and install packages for an offline Arch box"
echo " using another, online box as a proxy of sorts. So far, this only works if"
echo " the proxy box has bash, wget, tar, sed and grep installed."
echo " It's probably also entirely unsafe, incompatible and will destroy your box"
echo " in a fiery blaze if you try to use it (you know the drill)."
echo
echo -e " ${green}Options${dull}"
echo -e " ${green}=======${dull}"
echo " init Step 0 (as it were): take a snapshot of your system"
echo " fetchdb Step 1: download the packagelist database"
echo " fetch <file> Step 2: download requested packages"
echo " install Step 3: update your system"
echo " clean Step 4: delete any unneeded files"
echo
echo " howto More detailed instructions"
echo " bugs Problems and future plans"
echo " faq As it says"
echo " warranty (in case it ever becomes necessary)"
echo " version (to be honest, I wasn't expecting to go beyond v1-rc or so)"
echo " help You're reading it, doofus :^)"
echo
# dl_list Test the find_dependencies function"
# verint Test the compare_versions function"
esac

Answer 1

AnimaInvicta wrote:
Since my Arch box has no internet connection (yes, I'm that much of a masochist), I wrote a bash script to act as a proxy of sorts for pacman, at least as regards downloading. It took a while to get it working (intermittent web access and all) but here it finally is, as promised.
It works well for me, and I only hope that it will help anyone else in the same situation (if there is anyone else in the same situation).
The script has various options to explain everything you need to know, but here's a quick word of warning: it only works on POSIX-compatible computers (i.e. no Wintendows, usually) and the proxy box needs to have bash, wget, tar, grep and sed installed. I don't see this as being a problem, but if I'm wrong, tell me all about it.
#!/bin/bash
readonly VER="2.1"
readonly wgetVER=`wget -V|sed '1!d'`
# DEBUG levels:
# Set to 0 for ALERT-level messages only
# Set to 1 for INFO-level messages (I prefer this)
# Set to 2 for DEBUG-level messages
# Set to 3 for all messages
readonly DEBUG=0
# Set DEBUGLOG=1 to print ALERT, INFO and DEBUG messages to DEBUG.log
readonly DEBUGLOG=0
readonly COLOUR=true
if [[ $COLOUR == true ]] ; then
readonly red='\e[31m'
readonly warn='\e[31;7m '
readonly green='\e[32m'
readonly lgreen='\e[32;1m'
readonly yellow='\e[33;1m'
readonly lblue='\e[34;1m'
readonly dull='\e[0m' ; fi
# Sticky details:
# tar seems very slow (vfat? transform? USB?)
function bugspray {
[[ -z $2 ]] && verbosity=1 || verbosity=$2
if (( DEBUG >= verbosity )) ; then
(( $2 == 0 )) && intro="${warn}ALERT${dull} | "
(( $2 == 1 )) && intro=" INFO | "
(( $2 > 1 )) && intro=" ${yellow}DEBUG${dull} | "
echo -e "$intro$1"
(( DEBUGLOG == 1 )) && echo "`date +%F_%T` | $1">>log-alpps/DEBUG.log ; fi ; }
function compare_versions {
# Requires two ordinary, untweaked version strings, first local, second distant
if [[ -z $2 ]] ; then
bugspray "compare_versions didn't receive two variables." 2 ; fi
update='no'
[[ -z $2 ]] && return
local localversion=(`echo $1|sed -r 's/([[:alpha:]]+)/ \1 /g;s/[\.:~_-]/ /g'`)
local distantversion=(`echo $2|sed -r 's/([[:alpha:]]+)/ \1 /g;s/[\.:~_-]/ /g'`)
local max=$((${#distantversion[*]}))
bugspray "localversion: ${localversion[*]}; distantversion: ${distantversion[*]}" 2
for (( x=0 ; (( x < $max )) ; $((x++)) )) ; do
bugspray "local version part: `echo ${localversion[$x]}` | distant version part: `echo ${distantversion[$x]}`" 2
if [[ `echo ${localversion[$x]}|grep -E '^[0-9]*$'` && `echo ${distantversion[$x]}|grep -E '^[0-9]*$'` ]] ; then
bugspray "Numerical comparison" 2
bugspray "localver (${localversion[$x]}) less than distantver (${distantversion[$x]}): $(( 10#${localversion[$x]} < 10#${distantversion[$x]} ))" 3
# Different compare methods for numbers, single letters and multiple letters.
if (( 10#${localversion[$x]} < 10#${distantversion[$x]} )) ; then
update='yes'
break ; fi ; fi
if [[ `echo ${localversion[$x]}|grep -E '^[[:alpha:]]$'` && `echo ${distantversion[$x]}|grep -E '^[[:alpha:]]$'` ]] ; then # single letter -> compare
bugspray "Lexicographical comparison" 2
bugspray "localver (${localversion[$x]}) less than distantver (${distantversion[$x]}): [[ ${localversion[$x]} < ${distantversion[$x]} ]]" 3
if [[ ${localversion[$x]} < ${distantversion[$x]} ]] ; then
update='yes'
break ; fi ; fi
if [[ `echo ${localversion[$x]}|grep -E '^[[:alpha:]]+$'` && `echo ${distantversion[$x]}|grep -E '^[[:alpha:]]+$'` ]] ; then # string -> drop
# cvs / git / beta -> what are the rules?
bugspray "Sequence of letters. Not treating as version number." 2 ; fi
done ; }
function find_dependencies {
# Determine existence via unique path
bugspray "+ ${lgreen}Building download list for package \"$1\"${dull}" 1
local packagename
local dlfile
local package_found=no
if [[ `ls .temp-alpps/*.db/$1-* 2>/dev/null` ]] ; then
bugspray "Analogous package-name file(s) found" 2
for x in `ls .temp-alpps/*.db/$1-*/desc` ; do
bugspray "Checking $x" 2
packagename=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $x`
if [[ j$packagename == j$1 ]] ; then
bugspray "Exact match found: $packagename = $1" 2
dlfile=`sed '/%FILENAME%/,+1!d;/%FILENAME%/d' $x`
local distantversion=`sed '/%VERSION%/,+1!d;/%VERSION%/d' $x`
local package_full=$packagename-$distantversion
local the_repo=`echo $x|sed -r 's/.temp-alpps\/(.*)\.db.*/\1/'`
local the_path=`echo $x|sed -r 's/desc//'`
unplus=`echo $packagename|sed 's/\+/\\\+/g'`
local local_version=`sed -r '/^'$unplus' /!d;s/.* //' .temp-alpps/snapshot.state`
package_found=yes
break ; fi ; done ; fi
if [[ -z $dlfile ]] ; then
bugspray "Exact match not found -> looking for replacements." 2
if ! [[ -z `grep $1 .temp-alpps/*.db/*/depends` ]] ; then
local provisional=`grep $1 .temp-alpps/*.db/*/depends|sed -r 's/\/depends.*//'`
for package in $provisional ; do
local providence=`sed -r '/%PROVIDES%/,/^$/!d;/%PROVIDES%/d;/^$/d;s/>.*//;s/=.*//' $package/depends`
for y in $providence ; do
if [[ j$y == j$1 ]] ; then
# But what about when several packages provide the same thing and ALPPS picks the wrong one? Can it happen?
local newdep=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $package/desc`
bugspray " -> Package \"$newdep\" provides \"$1\"." 1
find_dependencies $newdep
break 2 ; fi ; done ; done
if [[ -z $newdep ]] ; then
bugspray "${warn}Warning:${dull} package \"$1\" not found in database." 0 ; fi ; fi ; fi
bugspray "package: $1 | package_found = $package_found" 2
# Add file to download_list if not up to date and if not already present
if [[ j$package_found == jyes ]] ; then
local already_got_one=no
unplus=`echo $dlfile|sed 's/\+/\\\+/g'`
[[ `echo $download_files|grep $unplus` ]] && already_got_one="yes" && bugspray "${green}Package in queue${dull}" 1
[[ `ls downloads/$dlfile 2>/dev/null` ]] && already_got_one="yes" && bugspray "${green}Package proxied${dull}" 1
compare_versions $local_version $distantversion
bugspray "compare_versions says: $update" 2
! [[ -z $local_version || $update == yes ]] && already_got_one="yes" && bugspray "${green}Package already installed and up-to-date${dull}" 1
if [[ j$already_got_one != jyes ]] ; then
bugspray "${green}Confirm downloading${dull}" 1
download_files+="$dlfile "
download_array[${dlfile}]=$the_repo
bugspray "Checking for further dependencies" 2
if [[ -a $the_path/depends && `grep -E '^%DEPENDS%$' $the_path/depends` ]] ; then
local dependency=`sed -r '/%DEPENDS%/d;/^$/q' $the_path/depends`
bugspray "Dependencies of \"$packagename\":$dependency" 3
for x in $dependency ; do
# Is this dependency already installed? Sort-of the same as higher up; it just saves time here.
local depname=`echo $x|sed -r 's/>.*//;s/=.*//'`
local depminver=`echo $x|sed -r 's/.*>//'`
bugspray "Dependency name: \"$depname\"" 2
if ! [[ `grep -E '^$depname ' .temp-alpps/snapshot.state` ]] ; then
find_dependencies $depname ; fi ; done
else
bugspray "${lblue}End of the line${dull}: package \"$packagename\" has no dependencies." 2
true ; fi ; fi ; fi ; }
function failover_fetch {
echo "Downloading `echo $1|sed 's/.*\///'`"
[[ -d log-alpps/ ]] || mkdir log-alpps/
[[ -z $success ]] || unset success
for mirror in $(<.temp-alpps/snapshot.mirrorlist) ; do
# If DEBUG>0, shouldn't redirect output
url=`echo $mirror|sed 's/$arch/'$arch'/;s/$repo/'${download_array[$1]}'/'`/$1
bugspray "Connecting to $url" 2
wget -o .temp-alpps/very-temp-log -U "Arch Linux Pacman Proxy Script version $VER / $wgetVER" -P downloads/ $url && success=true
cat .temp-alpps/very-temp-log>>log-alpps/download.log
rm -f .temp-alpps/very-temp-log
if [[ -n $success ]] ; then
bugspray "Download: \$success = true" 2
break ; fi ; done
if [[ -z $success ]] ; then
bugspray "${warn}FAIL:${dull} $1 not accessible on known mirrors." 0 ; fi ; }
case $1 in
warranty)
echo
echo -e " ${green}Warranty${dull}"
echo -e " ${green}========${dull}"
echo " This program is free software. It comes without any warranty, to"
echo " the extent permitted by applicable law. You can redistribute it"
echo " and/or modify it under the terms of the Do What The Fuck You Want"
echo " To Public License, Version 2, as published by Sam Hocevar. See"
echo " http://sam.zoy.org/wtfpl/COPYING for more details."
echo
howto)
echo
echo -e " ${green}How to use ALPPS${dull}"
echo -e " ${green}================${dull}"
echo -e " ${yellow}+ Step 0${dull}: prime the engine"
echo " On your offline box, copy the ALPPS script onto a removable medium, cd into"
echo " it's directory, then run it with the \"init\" option. This essentially"
echo " takes a snapshot of your system, package-wise. Your removable medium is"
echo " now ready for use!"
echo -e " ${yellow}+ Step 1${dull}:"
echo " On the proxy box the first order of business is to download and decompress"
echo " the current package lists. You can do this with the \"fetchdb\" option."
echo -e " ${yellow}+ Step 1 alt${dull}:"
echo " If you want to download the package lists without decompressing them,"
echo " \"fetchdb simple\" will do this. It's a lot quicker, but please keep"
echo " in mind that you cannot then do anything in step 2."
echo -e " ${yellow}+ Step 2${dull}:"
echo " If you want to download packages (and why wouldn't you?), the easiest way"
echo " is to prepare a simple text file containing the names of the packages you"
echo " want, one on each line. Run ALPPS with the option \"fetch <filename>\" and"
echo " sit back and wait until it's done. Don't worry about dependencies: ALPPS"
echo " handles them automatically."
echo -e " ${yellow}+ Step 2 alt${dull}:"
echo " As a convenience, \"fetch full\" will download all the packages needed to"
echo " update the offline box."
echo -e " ${yellow}+ Step 3${dull}:"
echo " Finally, back on your offline box, run ALPPS again with the \"install\""
echo " option. This will update the package list and copy the package files into"
echo " local cache. You can now run \"pacman -S <package names>\" to finish"
echo " installing the packages."
# ALPPS no longer installs the packages, just caches them locally.
# Not entirely sure why; it just felt too klutzy.
echo -e " ${yellow}+ Step 4${dull}:"
echo " You will now probably want to delete all the stuff you no longer need. Run"
echo " ALPPS with the option \"clean\". This won't touch your request file(s). It"
echo -e " ${red}WILL${dull} delete log files, so if you want to keep them, back them up first."
echo " Next time you do this, don't forget to run ALPPS with \"init\" again."
echo
bugs)
echo
echo -e " ${green}Where this goes wrong${dull}"
echo -e " ${green}=====================${dull}"
echo " + ALPPS is a bit slow when packages have many dependencies. It's a recursive"
echo " shell script: what did you expect?"
echo " + It only uses the settings in /etc/pacman.conf. If your conf file is"
echo " elsewhere, you're SOL. Similarly, the repositories are all taken from"
echo " /etc/pacman.d/mirrorlist. If you added any custom repositories in"
echo " /etc/pacman.conf (or any other file), ALPPS ignores them."
echo " + As it stands, ALPPS only works when the proxy box has bash, wget, sed,"
echo " grep and tar installed. The offline box needs bash, pacman and sed (in"
echo " theory, this shouldn't be a problem...)."
echo " + There is as yet no way of handling package groups, short of listing every"
echo " member of the group."
echo " + I don't think ALPPS will ever be able to handle AUR packages. Of course, if"
echo " you're compiling AUR packages, you probably won't need something like this!"
echo
faq)
echo
echo -e " ${green}Frequently Asked Questions${dull}"
echo -e " ${green}==========================${dull}"
echo
echo -e " ${red}Q:${dull} Can I run the whole thing off a USB stick?"
echo -e " ${lblue}A:${dull} Certainly."
echo
echo -e " ${red}Q:${dull} Can I use a non-POSIX computer as a proxy (e.g. Windows)?"
echo -e " ${lblue}A:${dull} Probably not, unless that computer has a POSIX-compatibility layer"
echo " installed (such as Cygwin) with bash, wget, tar, sed and grep."
echo
echo -e " ${red}Q:${dull} Do I need to download the package list, waste time updating my computer,"
echo " then go back again to download the packages I want?"
echo -e " ${lblue}A:${dull} Nope! You can update the packagelist database and download piping-hot"
echo " fresh packages, all in one sitting."
echo -e " ${red}Q:${dull} You mean I won't have to futz around with package lists that keep updating"
echo " ten minutes after I download them?"
echo -e " ${lblue}A:${dull} Exactly. Nice, isn't it?"
echo
echo -e " ${red}Q:${dull} Does this thing handle SSL and signed packages?"
echo -e " ${lblue}A:${dull} ALPPS uses SSL if:"
echo " - the mirror has an https address (at present, none do), and"
echo " - wget on the proxy box is compiled with SSL support"
echo " Concerning signed packages: no, but then again, it doesn't need to. ALPPS"
echo " downloads packages; it's pacman's responsibility to verify them. Your"
echo " system remains safe (or as safe as it ever was, at any rate)."
echo -e " ${red}Q:${dull} What about gpg keys? I need to get them."
echo -e " ${lblue}A:${dull} ...maybe later, say, in version 3."
echo
echo -e " ${red}Q:${dull} ALPPS says it \`prepares databases´. Can pacman still use them after this?"
echo -e " ${lblue}A:${dull} Yes, it can. In fact, what ALPPS does is extract the databases into a"
echo " temp directory, without altering the original .db files."
echo
echo -e " ${red}Q:${dull} Where can I contact you?"
echo -e " ${lblue}A:${dull} For constructive, useful questions and comments: [email protected] and"
echo " be sure to mention Arch Linux in the subject."
echo " For flames, trolling, spam and the like, visit your local bitbucket."
echo
todo)
echo
echo -e " ${green}What's next?${dull}"
echo -e " ${green}============${dull}"
echo -e " + I ${red}might${dull} be able to extend the reach of this thing to the AUR."
echo " Yes, I know I said ALPPS couldn't (in \"bugs\"), but I've learned"
echo " something new since then. It'll be tricky, though, since the AUR web"
echo " interface delivers results in python. While this should be a Good Thing,"
echo " bash isn't very good at handling python-format lists and dicts."
echo
-v|ver|version|--ver|--version)
echo -e "${lgreen}Arch Linux pacman proxy script${dull} | ${yellow}version $VER${dull}"
dl_list) # Debugging
(( DEBUG == 0 )) && echo "the \"dl_list\" option is only for debugging" && exit 0
arch=$(<.temp-alpps/snapshot.architecture)
unset download_files
unset download_array
declare -A download_array
find_dependencies $2
bugspray "Download list: $download_files" 3
echo "Download list v2:"
for x in $download_files ; do
bugspray " File \"$x\" from repo \"${download_array[$x]}\"" 3
echo " Full URL: http://mirror.archlinux.org/${download_array[$x]}/os/$arch/$x" ; done
verint) # Debugging
(( DEBUG == 0 )) && echo "the \"verint\" option is only for debugging" && exit 0
for x in `ls .temp-alpps/*/$2-*/desc` ; do
packagename=`sed -r '/%NAME%/,+1!d;/%NAME%/d' $x`
if [[ j$packagename == j$2 ]] ; then
bugspray "Unique package-name file found: $packagename=$2" 2
distantversion=`sed '/%VERSION%/,+1!d;/%VERSION%/d' $x`
dlfile=`sed '/%FILENAME%/,+1!d;/%FILENAME%/d' $x`
package_full=$packagename-$distantversion
the_repo=`echo $x|sed -r 's/.temp-alpps\/(.*)\.db.*/\1/'`
the_path=`echo $x|sed -r 's/desc//'`
local_version=`sed -r '/^'$packagename' /!d;s/.* //' .temp-alpps/snapshot.state`
break ; fi ; done
echo "full package filename: $dlfile"
echo "Comparing local ($local_version) and distant ($distantversion) versions of $2..."
compare_versions $local_version $distantversion
#compare_versions 2.5beta3-2 2.5cvs4-1
echo "...and the verdict is: $update"
init)
x=`uname -m`
if [[ -x /usr/bin/pacman ]] ; then
[[ -d .temp-alpps ]] || mkdir .temp-alpps/
pacman -Q>.temp-alpps/snapshot.state
sed -r '/^Se/!d;s/Server = //' /etc/pacman.d/mirrorlist>.temp-alpps/snapshot.mirrorlist
sed -r '/^\[/!d;/options/d;s/\[(.*)\]/\1/' /etc/pacman.conf>.temp-alpps/snapshot.repositories
sed -r '/^Architecture/!d;s/^.*= //' /etc/pacman.conf>.temp-alpps/snapshot.architecture
[[ ! -s .temp-alpps/snapshot.architecture || auto == $(<.temp-alpps/snapshot.architecture) ]] && $x>.temp-alpps/snapshot.architecture
sed -r '/^SyncFirst/!d;s/.*= //;s/ /\n/g' /etc/pacman.conf>.temp-alpps/snapshot.prioritypackages
echo "Current state recorded."
else
echo "You seem to be running ALPPs on an unsupported system."
echo "ALPPS is the ${lgreen}Arch Linux Pacman Proxy Script${dull} and simply"
echo "won't run properly on a non-pacman OS (except when proxying)."
echo "Aborting. Sorry."
exit 1 ; fi
fetchdb)
[[ -d downloads ]] || mkdir downloads
arch=$(<.temp-alpps/snapshot.architecture)
declare -A download_array
for repo in $(<.temp-alpps/snapshot.repositories) ; do
download_array[${repo}.db]=$repo
location="${repo}.db"
failover_fetch $location ; done
if [[ j$2 != jsimple ]] ; then
[[ -d log-alpps ]] || mkdir log-alpps
[[ -a syncfirst ]] && rm -f syncfirst
echo "Preparing databases for local processing. This might take a while."
for x in $(<.temp-alpps/snapshot.repositories) ; do
echo "Preparing $x.db"
if [[ -a downloads/$x.db ]] ; then
[[ -d .temp-alpps/$x.db ]] || mkdir .temp-alpps/$x.db/
tar -xzC .temp-alpps/$x.db/ -f downloads/$x.db --transform 's/:/§/' --no-same-owner && echo "$x database ready for use."
else
echo -e "Database $x.db is missing. Skipping. This will probably cause problems." ; fi ; done
# SyncFirst package warning
[[ -z $download_list ]] || unset download_list
[[ -a syncfirst ]] && rm -f syncfirst
for x in $(<.temp-alpps/snapshot.prioritypackages) ; do
find_dependencies $x ; done
bugspray "download_files: $download_files" 2
if [[ ! -z $download_files ]] ; then
echo "New version(s) of SyncFirst package(s):"
for x in $(<.temp-alpps/snapshot.prioritypackages) ; do
[[ ! -z `echo $download_files|grep $x` ]] && echo "- $x" && echo $x>>syncfirst ; done
echo "Package names placed in 'syncfirst'. Don't forget to download them!" ; fi ; fi
fetch)
for x in $(<.temp-alpps/snapshot.repositories) ; do
if ! [[ -d .temp-alpps/$x.db/ ]] ; then
bugspray "You must first download a fresh database with the \"fetchdb\" option" 0
exit 1 ; fi ; done
arch=$(<.temp-alpps/snapshot.architecture)
unset download_array
declare -A download_array
unset download_files
if [[ $2 == full ]] ; then
echo " Building full upgrade list. This will almost certainly take a while."
echo " Please wait..."
for y in `sed -r 's/^(.*) .*/\1/' .temp-alpps/snapshot.state` ; do
bugspray "Examining $y" 1
find_dependencies $y ; done
elif [[ -a $2 ]] ; then
echo " Determining dependencies. This might take a while. Please wait."
for y in $(<$2) ; do
bugspray "Examining $y" 1
find_dependencies $y ; done
else
echo " Error: you should provide a list-file or the keyword \"full\"."
echo " (see 'alpps.sh howto', step 2)"
exit 1 ; fi
if [[ -z $download_files ]] ; then
echo " No files to download: either none found, or all found are up to date."
else
[[ -d downloads ]] || mkdir downloads
for x in $download_files ; do
failover_fetch $x ; done ; fi
install)
dbpath=`sed -r '/^DBPath/!d;s/^.*= //' /etc/pacman.conf`
[[ -z $dbpath ]] && dbpath="/var/lib/pacman/"
bugspray "dbpath: $dbpath" 2
echo " Updating databases"
cp -ft ${dbpath}sync/ downloads/*.db
cachedir=`sed -r '/^CacheDir/!d;s/^.*= //' /etc/pacman.conf`
[[ -z $cachedir ]] && cachedir="/var/cache/pacman/pkg/"
echo " Caching packages"
cp -f downloads/*.pkg.tar.xz $cachedir
echo
echo " Don't forget to install/update your packages."
echo " Exercise proper caution."
echo " Have fun!"
clean)
echo "Cleaning up. This might take a while."
rm -fr .temp-alpps/ && echo "Buffer directory deleted" || echo "Problem deleting buffer directory '.temp-alpps/'"
rm -fr log-alpps/ && echo "Log directory deleted" || echo "Problem deleting 'log-alpps/'"
rm -fr downloads/ && echo "Downloaded packages deleted" || echo "Problem deleting 'downloads/'"
echo
echo -e " ${lgreen}Arch Linux pacman proxy script${dull}"
echo
echo -e " ${green}Description${dull}"
echo -e " ${green}===========${dull}"
echo -e " This is a bash script to fetch and install packages for an offline Arch box"
echo " using another, online box as a proxy of sorts. So far, this only works if"
echo " the proxy box has bash, wget, tar, sed and grep installed."
echo " It's probably also entirely unsafe, incompatible and will destroy your box"
echo " in a fiery blaze if you try to use it (you know the drill)."
echo
echo -e " ${green}Options${dull}"
echo -e " ${green}=======${dull}"
echo " init Step 0 (as it were): take a snapshot of your system"
echo " fetchdb Step 1: download the packagelist database"
echo " fetch <file> Step 2: download requested packages"
echo " install Step 3: update your system"
echo " clean Step 4: delete any unneeded files"
echo
echo " howto More detailed instructions"
echo " bugs Problems and future plans"
echo " faq As it says"
echo " warranty (in case it ever becomes necessary)"
echo " version (to be honest, I wasn't expecting to go beyond v1-rc or so)"
echo " help You're reading it, doofus :^)"
echo
# dl_list Test the find_dependencies function"
# verint Test the compare_versions function"
esac
I've always resorted to use VPN, then I started using cntlm, this looks very useful (and a lot of work on your part!) Thanks alot!
DoctorZeus