Re: Autoload from a web page?

[Jonas Bernoulli]

(what's up with me today. This also did not go to the list in the first try)

On Tue, Dec 29, 2009 at 23:25, Lennart Borgman
<address@hidden> wrote:
> On Tue, Dec 29, 2009 at 5:52 AM, Tom Tromey <address@hidden> wrote:
>>>>>>> "Lennart" == Lennart Borgman <address@hidden> writes:
> My intention is not to replace ELPA, but rather to create a backend
> (and partly a frontend) for installing/updating through the web from a
> repository (like bazaar on Launchpad).

The package manager should support multiple package repositories.

In theory this could mean that we end up with many competing repositories
and this is clearly something that should be avoided. Instead we should have
a few repositories with well defined purposes:

1. A "complete" repository.

This is what I am currently trying to do with the Emacsmirror [1]. This should
contain most packages that are available, regardless of quality and general
usefulness. It does not mean that packages could absolutely not be removed
or that it should not impose a minimal set of restrictions but the focus would
be on quantity not quality. (more on this below)

This would include packages that are broken and this would have to be pointed
out to users. I do think that it does make sense to include broken (or
very basic)
packages, because this allows potential new maintainers to easily find existing
implementations and go from there.

2. A "tested" repository.

ELPA could probably take this role. This repository should only contain packages
known to work with recent versions of Emacs and that are otherwise sound. QA
should happen here.

3. The FSF repository

This would contain only packages that are of could quality and generally useful
AND for which copyright assignments have been filled out.

I do not think that authors who release their packages under incompatible
licenses or do not want (or can't) assign the copyright to the FSF have bad
intentions.  Especially I do not think that because a package is still released
under the GPL-2 instead of GPL-3 is because they are against the move but
that they simply do not see much benefit in it or have forgotten to do it.

I have released all my Emacs extensions under the GPL-3 (at least I think so)
and do not think that it does not make sense to use any other license.  However
authors of packages that are only updated when bugs are reported might just not
see the need (or just forget about it) to update the license.

Having a "complete" repository is in my eyes not against the notion that we
want authors to move to the GPL-3.  Instead they might actually relize that
their packages are only included in the "complete" repository a long with many
clearly inferior code, not because of it's quality but because if
licensing issues.

I do not know what ELPA's position on license issues are. So if they include
packages e.g. releases as-is (and especially it includes package still released
under the GPL-2 (which I expect)) so it might be necessary to add an addtional
repository inbetween 2 and 3.

However I do strongly believe that it would be very unproductive to have more
than four repositories. So e.g. I do not think that it would make any sense to
create a repository for packages where the license is good but that are rubbish.

> Currently I have written to different ways to do this:
>
> - Download a whole package of elisp files.

This could be done in two ways.

- Download an archive
- Clone a dvcs repository

I favor the model where the package manager does clone the repository from
the repository - ehm ... Okay I have used the terminology used by other in
this thread so far. But actually I think we should talk of the repository as
a dvcs (git, bazaar) repository containing the history of a particular package
which is included in a mirror (Emacsmirror, ELPA) do avoid confusion.

And while we are talking about terms. I have used the term package to mean
the thing installed by the package manager.  Of course many packages
consist of only one library (the thing providing a feature) but
nevertheless these
are to different things.

So again:

I favor the model where the package manager does clone the repository from
the mirror as one of the steps required to install a package.

This would mean that the dvcs used by the mirror does have to be installed
on the users machine. So if this is not desirable archives could be created on
the mirror from each tagged release. For packages where upstream does not
use a vcs an archive could be created for each version where the library of
the "mainfile" has changed.

Creating the archives on the mirror would not be to hard. But adding support
in the package manager for installing either from a repository OR from an
archive would complicate things. So unless there are strong arguments
against it I would favor the vcs approach a lot, as it adds at least
the following
benefits:

- When a user installs a package for the first time from an untrusted source
e.g. the Emacsmirror he should read the packages source which is a
considerable amount of work. (I will comment elsewhere on how the installation
process would have to be adjusted to allow this.)

But provided the user verified the integrity of a package when he
first installed
it, the use of a vcs would make it much easier to only verify the difference to
the installed package once it is updated.

- The user can use the same interface (the package manager) used to find and
install working packages in order to clone broken packages, fix them, then
install them, and push his own fixed branch (and of course this should be
done in a way that makes it easy to also inform upstream of the changes,
either to have him merge the changes or to take over as a maintainer.

> - Download a package part by part as the elisp files are needed (kind
> of autoloading and installing locally through the web).

I do not like this idea at all. As I have said (and as you are well aware) there
is a difference between package and libraries (and supporting files like README,
or images).

The package manager should only operate on complete packages.

> Those work currently in the only conventient way I could find out at
> the moment. They download files directly from the web interface of the
> repository (currently only Launchpad is supported). This has the
> benefit that no sw except for Emacs is needed. From the user's point
> of view it is easy.

As I have said above I favor the vcs approach. It it is not feasible to clone
repositories then archives should be offered for download (which isn't hard).
Fetching seperate files seam backward and it actually complicates things
and as far as I can tell it has no benefits whatsoever.

Also while the Emacsmirror hasn't been integrated with a package manager
yet I do think that it already now makes the process of installing packages
(and especially updating) much easier than to fetch packages from all over
the internet.

And this is precisely because it uses a dvcs. You simply clone the desired
package and from that on you simply have to run "git pull" to bring your local
copy up-to-date. You do not have to remember where you got the package.

This is especially useful for packages that are distributed upstream as
versioned files (that is with a version suffix) dumped on a webpage.  The
task of finding the most recent version is already handled by the mirror -
otherwise you would have to go to the homepage, locate the most recent
link and the download it. And that for all installed packages.

(And yes of course while the Emacsmirror has not been integrated with
a package manager yet you will have have to make the package known
to Emacs manually. There also is not automated way for your local QA
yet, but "git diff OLD_HEAD.." make this much easier.

> There is no support for this use of the repository (bazaar) web
> interface so this use is a bit fragile. I imagine ELPA could be
> something in between giving this support

I agree as described above.

> (for example by simply taking
> a snapshot from the repository and placing that on the web as just
> directly accessible files).

I also agree but only if the snapshot is a archive. (I also think that it is
possible to download individual raw files from lp but haven't checked as
I am not interested in that.)

> ELPA could be the trusted source of where to begin, a kind of
> directory for resources to download this way. An end Emacs user should
> be able to trust ELPA that the sources are correct and virus/trojan
> free (at least).

Yes. But there should also be the Emacsmirror which isn't about trust
but completeness.

> The second form, ie downloading part by part, depends simply on a file
> with autoload information for the package of elisp files. ELPA could
> produce such a file (or the package maintainer of course). (I have the
> tools for that.)

I also have a tool for this (`elx.el' [2]). I would like to see yours - maybe we
can merge the two.

> Downloading part by part naturally puts some more rigid restrictions
> on the elisp files in the package than normally, but it looks
> solveable to me. (I am struggling with the elisp package nXhtml to
> make it fit this right now.)

I do not understand what nxhtml has to do with it, after all the headers
in emacs lisp libraries are not in html/xml.

> Downloading part by part has some advantages and disadvantages:
>
> + You download (automatically) just what you need and use.

And when you have no connection to the internet you are stuck with
what you have.  Again the atom a package manager should operate on
are packages. (Then again it might make sense to create seperate
packages for libraries found in the contrib directory of larger packages.)

> + To update you simply delete the local file.

Deleting a directory containing the complete package isn't much harder.
You just have to add "-r" to your "rm" - in fact it is easier, as "rm -r foo" is
saver than "rm foo*" which might also remove "foo-bar" ;-)

> - Files could become out of phase. Some version control should be good.

Yes exactly.

> - You have to be connected to the internet when you want to use a
> feature you have not used before.

Yes.

[1] http://www.emacsmirror.org
[2] http://github.com/tarsius/elx