Pacman Development

Question

Hi all,
Im new to Arch Linux, and so far im loving it and would like to help with its development.
I have little knowledge of the in's and out's of ABS and Pacman but after alittle research on them I would like to discuss how the current implementations work, what the plains are for the future, and to give some ideas too.
Pacman,
I have read that the current system uses a file based package management, which *could* be a problem with scale and speed. I have read many comments on moving to a DB system, here are my comments;
A database system is a very good choice and should improve both scalability and speed issues. A database system could create opportunities such as more complex searching, "pacman" history, rollback features, etc. One problem mentioned on a database system is the "ease" for people to customize the database, create packages etc. With the FUSE http://fuse.sourceforge.net/ we can mount the database as a file system which is similar to its current layout allowing people to edit the database in the current methods (bash scripts etc)
I would recommend Sqlite as the database system as it has a large community, well documented and many other packages use Sqlite which can create opportunities for pacman to be used with other projects. However, i am unaware of any other database systems that can be compiled into the project (to save a dependency if needed), thats lightweight fast etc..
Once the information is in a database, We could infact take the whole pacman project "online", and provide a webservice to search the pacman database, this has the added advantage of being 100% uptodate, only returning information that your system requires, and altho a server would have additional cpu overheads of providing the search, I predict that because we would only be sending required information and not "everything" we would infact lower bandwidth. Hopefully this would make everything much more scalable.
However, if someone did want an offline cache of the database, I would recommend zsync (http://zsync.moria.org.uk/), I havent looked up its license yet, but Zsync as a utility is much better than rsync. zsync provides excellent features for single file sharing, requires nothing but a http server. zsync supports compression (into blocks with gzip) checksums, mirrors, etc.. There maybe a more efficient Database syncing tool which maybe more appropriate, comments welcome.
OK well i think thats it, I would love to know some more indepth details on how pacman works, such as language its written in, and current development of it, so please reply!
Mike

Answer 1

STiAT wrote:
phrakture wrote:
cactus wrote:
This has been talked about to death many many times over.
Search the forums... it is in there...
Search - pacman+sqlite
sqlite is also crap... you could probably find me saying that about 40 times in that search 8)
I totally agree on this. I've driven tests (not with a fully-imported pacman database, but with a similar relational structure, with about 3000 imported datasets), and realized that it hasn't been a lot of faster than pacman on ext3, since i had to query the database often, and also i had to compare the values. So just the access (fopen/stat) could be slower, but i doubt that fopen/stat of some hundret files takes longer than actually importing a (not that small) sqlite db properly, holding more values than i actually need.
Cutting down the file number of the files pacman uses actually could increase the speed a lot once more.
// STi
heya
from what i understand so far, pacman currently search's /var/lib/pacman/[extra|community|current|local] for any directories which match the search criteria. Pacman then searches /var/lib/pacman/[extra|community|current|local]/*/desc for another match.
The kernel is infact caching every file apart from "depends". Maybe the kernel is better at caching than a database? I suspect that a database driven would not have a great deal of speed benefit at the current package size of Arch, I believe tho that a database driven setup would provide easier future features such as linking ABS and Pacman closer together, So you could be able to specify a specific configuration of a package (Enable/Disable), and pacman would automatically know that this package must be compiled for your needs. If however you decide to install an extra package which maybe usable in another program, pacman can ask you if you wish to recompile/upgrade.
e.g.
If say i install bmpx, which has these options..
aac alsa cdparanoia debug flac ffmpeg hal mad modplug musepack nls ofa ogg oss p2p python sid theora vorbis
If i specifically want only alsa, pacman would compile bmpx rather than use binaries, and would ignore all other dependencies it wants to install (such as ogg, hal, sid, ffmpeg, etc).
If then, at a later date, i install beagle which uses HAL, pacman could see i have a package compiled without hal support and can offer me the choice to recompile to include it.
It would hopefully work the other way round too, which is where it gets most interesting. I have many-a-time been forced to uninstall EVERYTHING (on rpm and deb based systems) because there is a new low-level library update (libc or something). This cascading effect has broken my system countless times, a mirror would fail or a package wouldnt install and its too late, its uninstalled everything already!
Has anyone else had this problem?
As this mainly happens with libraries, and as far as im aware you can have multiple versions of libraries, what would be far better is to install the new version of the library, keeping the old one if a program still requires it. Once all packages have been updated or recompiled to use the newer library, then the old one can be safely removed.
Mike