Powerpill - A wrapper and download accelerator for Pacman

Question

I've moved all the powerpill info to this page: http://xyne.archlinux.ca/info/powerpill
Please continue to use this thread for discussion about powerpill.
Last edited by Xyne (2008-11-13 07:20:17)

Answer 1

The 404 errors are probably from mirrors that are out of sync with your top mirror (the one that pacman loads the database from).  Some mirrors won't yet have the latest packages yet. They should still get downloaded from another mirror though, because the metalink itself contains the mirror that was used to sync the database. If you notice that certain mirrors always throw 404 errors, you might consider removing them from your list (it's an indication that they don't update often enough). Also, take a look at "reflector" linked in my sig.
Actually, I just checked http://users.archlinux.de/~gerbra/mirrorcheck.html and holmes.umflint.edu hasn't synced in 16 days, so I think you can remove that one from your list for now.
I'd need to know what command you invoked powerpill with to get that pacman error ("powerpill -S" perhaps.... which leads to the same error as "pacman -S"). During a normal sync operation, powerpill invokes pacman 3 times:
     First with the pacman arguments that you passed it, adding the argument to print the package URIs.
     Once with -Ss to build a package-to-repo hash.
     Finally with the original arguments to pass control to pacman for installation.
Of course, there are sanity checks on which arguments were passed. If the requested operation does not involve downloading, control is passed to pacman immediately. If the requested operation included the "download only" option, it exits instead of calling pacman once downloads are complete. Powerpill doesn't play pacman-babysitter though, so it doesn't catch invalid pacman operations (although it does strip out any powerpill-specific arguments).
Aria is always invoked once and only once if the operation involves downloading, regardless of the number or size of the files.
Powerpill passes the pacman arguments it receives to powerpill along with the argument to print the URIs for the requested operation. Powerpill then takes this list, which contains the URIs to the files on your top mirror, and parses them to create a list of the pkgs that it should download. It then grabs the other mirrors for each repo from the pacman configuration files and builds a list of URIs to each requested packages on the different mirrors. These lists are used to create a single metalink for all the packages which is then piped to aria2c. This is done regardless of the number of files or their sizes. The single metalink makes it possible to download multiple files simultaneously and large files in segments. The metalink file itself contains information to determine how many connections to use for each file, so smaller files are not segmented. Even if there is only one small file to download, it is still just as fast as wget or curl or any other downloading utility.
When downloading files in the metalink, the list of URIs to the package on different mirrors is available for all packages, regardless of the connection limit per file. Aria2c chooses a mirror from the list and tries to download the package. 404 errors will occur if that mirror doesn't get have the latest version of the package. In that case, aria2c will just use another URI from the list until it gets the file. Each list will contain a URI to the file on the mirror that pacman used to sync the database, so that file will get downloaded from there if none of the other mirrors have the package yet.
Take a look at the metalink file generated for different operations.
powerpill -S <pkgA> <pkgB> <groupX> --get-metalink -
The "resources" tags contain the "maxconnections" attribute. You can confirm that small files are limited to 1.
My current version control system consists of leaving older versions in a directory on that server. Because of powerpill's "simple script" origin, I haven't given much thought to anything more serious, although I'll consider it. Thank you for your offer.
edit
Sorry about repeating myself within this post.
Last edited by Xyne (2008-10-12 19:55:53)