Package diffs

Question

I've seen quite a few requests by people on dial-up connections for a method to download diff packages instead of full ones.   
Since I have a high speed connection, I havent been too concerned about it, but I am about to roll-out a bunch of new machines and started thinking that downloading diff's of packages might be a good idea after all.
So I thought about it a little bit and have come up with a solution that will work on top of what is already there.  No changes would be needed with pacman.   In fact, I've messed around with it for the past couple of hours and am almost done.
Just for the proof-of-concept, I've written it in perl, but if people actually start using it, I might re-write it in C.   
Basically, this is what it does.  There are two parts to it. 
The server Part
I have a perl script on my webserver that runs ever half hour and checks the main ftp.archlinux.org/current and /extra repositories.  It keeps track of each package in there.  When a new package is updated, it downloads it.  Then it looks in one of the other mirrors for the old package that is being replaced, and downloads it.
Then it extracts each package and compares it file by file.  Any file that is new or updated is put in a new folder.  It makes a list of all the new, same, updated, and deleted files.  It then packages all this up into a diffpkg.tar.gz file and stores it on the webserver ready to download (if it is at least 10% smaller than the actual package).  This part is already done although I havent put it on a 30 minutes schedule yet.  Also note, this it is copying any updated or changed file directly.  It is not actually making a diff of each file that is different.
The client Part
The client part is another perl script that would be referenced in the XferCommand setting in /etc/pacman.conf so that it is run first when attempting to download a file.  It will get the package name from the file url, and will look in the users cache to see if there is an older version of the package there.  If so it will try to download a diffpkg from my website.  If there is no diffpkg file there, then it will use wget to download the file using the original url that was passed to it.
If it does get a diffpkg file, then it will open it up, check the pre-requisite package that it was generated from to see if that is the one in the users cache.  If it isn't then it will download the original URL (this could mean extra downloading if the packages are not installed in sequence, but I may have a solution for this).   
It then extracts the cached package into a temporary folder, extracts the diffpkg into another folder, and processes the deleted, updated and new files copying the content into the new package folder.  When it is done, it tarzips it all up and you should now have a package in your cache created from the diff set that is very similar to the package that would have been downloaded in entirety.   Pacman would then install the package as if it had been downloaded.  It shouldnt care if it was downloaded directly or if it was generated from the diffpkg.
Of course, this will only really be of benefit for those package upgrades that really only change or add a few files.   
I just tested this on the neverball package that was just updated.  The old pkg was 9mb, the new pkg was 14mb, the diffpkg was 12mb.  So it would only save you 2mb of download.   Not really a whole lot, but to some people it may be helpfull.
Thoughts?  Questions?  Anyone interested in testing it out when I'm done?

Answer 1

sarah31 wrote:this has come up several times and there was no satisfactory solution. it requires a bit more work on an already stressed developer core.
Right, because the thought was to change the existing system to accomodate a diff system.  My idea however, is to add-on to the existing system.  It is automatic, and does not require a developer or maintainer to produce the diff set.
sarah31 wrote:my problems with diff is that you would have to have the contents of the diff available for view. otherwise you could have difficulty convinceing people they are secure.
I'm not sure what you mean.   The files in the diffpkg.tar.gz file is on the webserver, and the contents can be viewed easily. 
sarah31 wrote:how would you do this?
sanity checks on diffs would be much more difficult on diffs i would think.
Oh, I think I see what you mean.  The files in the diff archive, are not diffs themselves.  They are the entire file, unmodified.  the difference between the package and the diffpackage is that the diffpackage is stripped of the files that were the same from the previous version of that package.
I figured that creating a package of diff files to "patch" existing files on the system, would not work because you never really know what files the user has modified unless you can't tell in advance which files are config or editable files and which ones arent... therefore putting a lot of hassle on maintainers and such.   So I figured why not create a system that compares the old package file against the new package file and just tarzip up the files that have changed or added.
so if you install widget-1.0.1-3.pkg.tar.gz and still have it in your /var/cache/pacman/pkg directory, then a new version widget-1.0.1-4.pkg.tar.gz comes along.  This application downloads widget-1.0.1-4.diffpkg.tar.gz and uses it to build widget-1.0.1.4.pkg.tar.gz locally from the old one in the cache.
There are some big limitations in this situation.
Firstly, if the new version of the package is re-compiled with different options, then all binaries in the package will likely be different.  Therefore a diff set would not be of any use, unless there are other files in the package that were not modified.
Secondly, If the user has cleared their cache, then they wont have the local file to build apon.
and Thirdly enough files need to be duplicated (and removed) to produce a savings in download size.   So I am thinking that some of the larger ones might be suitable, but certainly not all packages.
So I am guessing that this diff system would only be of benefit to some people, most likely on a slow dial-up internet connection, and it will only really be of benefit on the very slight updates to packages.   And since some people would be willing to use bzip to gain an extra little bit of compression to save download time, they might be willing to use this system which would probably only get an equivilent boost.