Re: [Chicken-hackers] Distributed egg repo proposal

[Felix]

Hi, Imran!

Many thanks for this detailed information. I'm personally not overly
familiar with portage, but what you suggest is very interesting and
sounds like an approach that could be made to work.

Please allow me to bring up the problems I see with this approach (or
with every future proposal). It's not intended to blindly bash your
suggestions, but to simply show up things that we should take into
account when thinking about... THE SYSTEM.

> In essence, we're trying to provide a single "blessed" tree of eggs.
> In itself, this doesn't have to be in a VCS (although thats a useful
> tool to maintain the tree in). All thats relevant is that, at the end,
> we have a central, blessed tree which allows us to install eggs and
> their respective docs - without potentially breaking because some
> upstream repo's are offline.

(constantly #t)

> We have the worst of both worlds right now. We're sort-of forcing all
> egg contributors to use the existing svn repo - with all the hassle
> that entails (getting commit access, importing sources, etc), while
> also having to deal with more than 1 place for a bugfix to be pushed
> to (so having that central repository doesn't really help, in the long
> run).

It's actually working quite fine. That patches will not flow quickly
enough to upstream is indeed right, but I don't see that as a critical
point. It is more important that the egg tree is in a working state:
there are many more users than contributors and it has to scale into
that direction as well. Communicating patches upstream is more a
problem of the responsiveness of the original maintainer, not so
much of the system that stores our eggs, IMHO.

> Why a local tree? Because this makes it easier to overlay a personal
> tree on top. In ~/.chicken-install, I can define a local overlay to be
> applied on top of the official tree. I can pop my new egg recipes in
> this local tree, and try them out, before submitting them upstream.

When mixing trees that way, how do you ensure the combination of them
is consistent?

> chicken-install first looks in *local-pkgs* and then *pkgs* to see if
> my-wonderful-egg-2.0.tgz exists. If it doesn't, then chicken-install
> looks upstream (from upstream-pkgs). If chicken-install had to fetch
> something from upstream-pkgs, then this is automatically copied to
> *local-pkgs*. This gives rise to a number of possibilities:

How can we be sure the system where the egg is to be installed
has tar/gunzip? (a pure mingw32 build doesn't have this).

> * we can have "recipe-my-wonderful-egg-latest", where chicken-install
> grabs the sources from the upstream repos HEAD.
> 
> * we can have "recipe-my-wonderful-egg-felix-2.0", pointing to Felix's
> forked version of my egg (different info for "upstream")

Forking eggs brings up the consistency problem again. I don't want
to fork, there should be no forks, I'd say.

> Now its much easier to track the real source history of any egg:
> 
>     chicken-install --upstream-checkout
> 
> So, without going through the hassle of forcing egg authors to push
> their sources to an egg svn repo (and either abandoning their existing
> VCS, or keeping their personal VCS in sync with the egg svn), we still
> have easy immediate access to the entire history of their egg's source
> code.

How would that be working? Would this require the VCS that holds the
code to be installed?

> Felix (who has unfettered write access to the entire tree) adds a
> patch to the patches/ dir, creates a new 2.0-r1 (or 2.0.1, as long as
> we're consistant) recipe copied from 2.0, and adds the following:
> 
>     (get-files
>       ...
>       (apply-patches "fix-for-chicken-4.9.patch"))
> 
> chicken-install will look under patches/ for
> "fix-for-chicken-4.9.patch", and apply it to the sources it already
> got.

That will require a patch tool on the client system, if this is
done by chicken-install.

> 
> We can get far beyond simple patches here, and maybe add other rules, like:
> * (apply-rename ) which will do a search & replace from one deprecated
> function name to a new one.

I'm not sure this can be reliably implemented, in the face of syntax
and modules.

A final note: Such a system as you describe (or the many possible
variations of it) will be of considerably complexity and will need a
long testing phase. This is not a point of criticism (any reasonably
smart system will be complex anyway), but we should be aware of the
fact that it will require manpower, and developers who are willing to
put a significant time into making this work - on all platforms. Do we
have the resources for that?

Thanks again for your input. Please take a moment to appreciate the
excessively careful tone of this reply, something that is quite
untypical for me. :-)


cheers,
felix