Distrib -e "Arch(org|code|pkgs|aur|forum|wiki|bugs|.*)?" -- thoughts

Question

design the output of every tool we use to look like merge-able chains in a DSCM.
this is something i've been thinking about long before i came to Arch.  i want to see next-generation package/configuration/change management in a distributed distribution.
I am familiar with git, and most of what i have tried is relating with it; however. it's only because i know much about it.  other possibilities would be bazaar?/mercurial/fossil/etc.  i like fossil; i have not tried it but it looks closest to what i want to achieve.  i don't think it could scale to the levels we'd need however.
ASSERTIONS
) all PKGBUILD are DSCM (git/?) based
) bugs should ride along with software, and be merge-able when branches merge
) cryptographic signatures for each user
) wiki for each software
) forum "channels" for each software
) P2P sharing of SCM (blobs/trees/commits in git) units
) P2P sharing of common SCM (packs in git) pack
) P2P sharing of user configs and ABS build trees; each user may host their own binary/source repo, and sign their packages)
) P2P and distribution are good
essentially, everything is a branch/tree and we use facilities of DSCM with a P2P layer above.  the arch servers could become another node in the system and a long term record keeping peer.  others could add servers.  you could open the wiki/bugs/etc offline, in a web browser, and merge later.  when you edit your PKGBUILDS, they can be forked by others and improved, maybe pushed to the core/community repos.  official repo builds could be signed by an official Arch GPG key.  bring everything as close to source as possible, and spread it out.
this is completely brainstorming right now, but i have done some tricky cool stuff with git.  i want to keep all/most of the logic/information withing the git DAG (commit graph).  i think we could do neat stuff with the git index, git grafts, and several operations could safely be done in parallel.  we could do mapreduce type calculations on the "ArchNet" to get crazy statistics and visualizations.
i intend to actually build something soon-ish-awhile.  right now im working on an app that can produce 3D visualizations in VPython from any kind of input stream... i want to hook that kind of stuff up and visualize the arch/linux/gnu/buzz.
another offshoot project for me was to use VPython (that app is really fun) to navigate and manipulate git repositories in real time.  imagine visualizing your system in 3D while working on it.  like a 3D admin panel where you overlay others configs and entire systems on to your own to see what changes/etc. DSCM can do this.
thoughts?  what other kinds of things could we do if everything Arch behaved like a P2P super-repository?
Last edited by extofme (2010-02-14 01:34:37)

Answer 1

Anntoin wrote:Some interesting ideas. But you need to start small first and make a proof of concept before you try and tackle everything. A detailed plan of how packages are handled and a basic implementation would be a start for example (still not a small job though), then testing the behaviours that you are interested with that framework. You have an idea where you want to go with this but you will need to focus on a few core features and show their benefit before anyone will consider this.
ah yes of course.  the first step is getting a distributed index, and a "package" format (packages become fuzzy items, below/above); this will be realized in the form of:
"AUR3 [aur-pyjs] implementation in python (pyjs) + JSON-RPC"
https://bbs.archlinux.org/viewtopic.php?pid=823972
i have a package in the AUR for that [aur-pyjs], but it's old as i haven't been able to update in awhile, and won't be until i secure a development job in my new city (next week hopefully).  aur-pyjs will be built on top of the concepts i have outlined in this thread, and will in time become prototype.  check it out; pretty neat even though it can't do much yet :-).  soon though, i will update the package, and it will then be able to run as a native python desktop app (pyjamas allows the same code to run as a website or a desktop app).  at that point, it will be trivial to implement connectivity to the old AUR/repos, and we will in effect have a pacman+aur replacement.  from there i will tackle bugs+forum, of which there are already several implementations on top of DSCM sub-systems to research and learn from.
stefanwilkens wrote:
Dieter@be wrote:Interesting ideas, I think i like them.
but trying to store too many big files (ie package files) inside a VCS seems like a bad idea.  space requirements will be much bigger then what we have now, unless you make the VCS "forget" about older versions or something...
mostly this.
the base of what you're proposing is a tremendous amount of data that would never be touched after a new version is released, how do your suggestions fit the rolling release model. Especially relatively large packages updated with high frequency (nvidia binary drivers, for instance) could cause the space requirement to increase rapidly unless moderated.
packages are not stored in the DSCM, their contents are.  the package itself is simply a top-level tree object in git, linking to all other trees and blobs comprising the package state, and a reference to said tree.  this means everything and anything that is common between _any_ package and _any_ version will be reused; if ten unrelated packages reference the same file, only one copy will ever exist; blobs are the same.  however, some packages may indeed create gigantic, singular blob type objects that always change, and this will be addressed (next...).
git compresses the individual objects itself, in gz format; this could be changed to use the xz format, or anything else.  it also generates pack files full of differentiated objects, also compressed. it would not always be necessary to have the full history of a package (if you look somewhere above, i breifly touch this point with various "kinds" of packages, some capable of source rebuild, some capable of becoming any past source/binary version, some a single version/binary only, etc.).  you would not have to retain all versions of "packages" if you did not want, but you could retrieve them at anytime so long as their components existed somewhere on the network.  servers could be set up to provide all packs, all version, effectively and automatically performing the intended duty of the "arch rollback machine".  the exact mechanism is not defined yet, but it will likely involve some sort of SHA routing protocol, to resolve missing chunks.
git's data model is stupid simple; structures can be created to represent a package, it's history, it's bugs/status, and it's information (wiki/etc.), in an independent way so they do not depend on each other, but still relate to each other, and possess knowledge of how to "complete" and find each other.  it will not be structured in the typical way git is used now.  unfortunately this is very low level git stuff, and difficult to explain properly, so i won't go there; just know that ultimately the system will only pull the objects you need to fulfill the directive you gave it, and there will be rules to control your object cache.  your object cache can then be used to fulfill the requests of others; ie. P2P.
since git itself is in a rather poor state when it comes to bindings, i will be using the pure python git library, dulwich, instead.  while in time this could be changed to use proper bindings, or some bits written as C modules, it's possible pypy will make all that unnecessary.  i don't need anything git core offers except its data structures and concepts; although, i intend to make the entire system (adding bugs/updating packages/editing wiki/editing forum/etc.) _completely_ 100% accessible from a basic git client.  for example, you could write a post in the forum by "committing" to a special branch; you could search the entire wiki, and its history from the terminal while installing; you could add a bug, and link a patch to it, directly usable and buildable by others for testing; this could all be done offline, and pushed once a connection was available... this will lead to all sorts of interesting paths...
in one super run-on sentence:
i intend to construct a "social", 100% distributed distribution platform, where everyone is a [potentially] contributing node and has [nearly] full access to all informations, each node's contributions are cryptographically verifiable, each node may easily participate in testing/discussion or lend computing resources, each node may republish variations of any object or collection under their own signature, each node may "track" or "follow" any number of signatures with configurable aggressiveness (no such thing as "official repos"; your personal "repo" is the unique overlay of other nodes you trust, and by proxy some nodes they trust; "official repos" degrade into an Arch signature, a Debian signature, Fedora, etc.), and finally, do all of this is a way that is agnostic to the customized distribution (or other package managers) above it, or it's goals, and eventually spread to other distros, thus creating a monstrous pool of shared bandwidth, space, ideas, and workload, whilst at the same time converging user/developer/tester/vendor/packager/contributor/etc. toward: person.
piece of cake
C Anthony
Last edited by extofme (2010-09-11 05:23:46)