Re: Interoperation between package managers

[Radon Rosborough]

> My view is that <pkg>-pkg.el should be built from metadata stored
> elsewhere (that's what we do in elpa.git).

That contradicts common usage, as reflected in the following Emacs
packages which ship a <pkg>-pkg.el file containing their metadata:

$ find . -name '*-pkg.el'
./cucumber.el/feature-mode-pkg.el
./ecukes/ecukes-pkg.el
./elisp-finalize/finalize-pkg.el
./emacs-async/async-pkg.el
./emacs-eclim/eclim-pkg.el
./ESS/lisp/ess-pkg.el
./evil/evil-pkg.el
./haskell-mode/haskell-mode-pkg.el
./markdown-toc/markdown-toc-pkg.el
./multiple-cursors.el/multiple-cursors-pkg.el
./skewer-mode/skewer-mode-pkg.el
./smartparens/smartparens-pkg.el
./zone-matrix/zone-matrix-pkg.el

The current manual [1] implies that it is the responsibility of the
upstream package maintainer to create a <pkg>-pkg.el file when
packaging their package. If that wasn't the intention, the
documentation should make this clear. But at this point, I would
advise against trying to enforce a new convention, given how useful it
is to have a consistent one established.

> I was thinking about it in the context of packages which can come
> from the user's config as well as from the system (i.e. installed by
> the sysadmin).

In this case, wouldn't the multiple versions be installed in entirely
different directories anyway?

> There are other cases where it can make sense. E.g. have both
> foo-1.0 and foo-2.0 installed at the same time, because foo-2.0
> requires Emacs-25 and you use both Emacs-25 and Emacs-24.

Again, in package.el, what you say makes perfect sense. But in a
source-based package manager, it seems better to handle this by having
the package manager check out the correct revision.

> I agree that these needs aren't the most common ones, but you don't
> gain anything by disallowing them, really.

I think you gain something by enforcing a simpler directory naming
convention. But of course this is still only relevant to source-based
package managers, and not to package.el.

> These look like info about the internals, and indeed we don't
> document them very much. Some of them are purposefully not
> documented ("use the source, 'cause it might change"). But there's a
> a lot of room for improvement anyway. I'll see if I can improve on
> that.

Sounds good. I agree that these things are sort-of internal, but the
problem is that users frequently encounter problems with package.el
("try deleting your ~/.emacs.d/elpa directory"; "try running
package-refresh-contents again"; "you put package-initialize in the
wrong place"), so it's important to have documentation that allows
debugging such problems. IMO, the package installation and loading
process doesn't need to be complicated. Everyone should be able to
understand it.

(In other words, "it should only be a black box if it always works".)

> AFAIK this is documented somewhere.

The "somewhere" part is a big part of the issue. If I want to know
something about package.el, maybe I should go to the Packages section
of the manual, or maybe the Packaging section, or maybe elpa.gnu.org,
or maybe ELPA's README, or maybe the Commentary in the source code of
package.el, or the docstrings, or the comments ???

This is very confusing for a new user who only wants to know why their
packages aren't being loaded.

Regarding the format of packages on the server, I think this
documentation is supposed to be at [2], but it doesn't seem to
actually be there.

> Actually "adding a directory to the load path" is only there by
> accident: it should be done by the autoloads file.

Does that mean M-x update-directory-autoloads will start inserting
load-path manipulation calls into the generated autoload files? Or
should package.el roll its own autoload generation routine?

> > * Why is package-initialize so slow?
>
> I don't know that it is, so I can't answer the question.
>
> >   What are the bottlenecks when you
> >   activate lots of packages?
>
> I haven't looked into.  If you find performance problems,
> M-x report-emacs-bug is probably the best option, so we can try to find
> how to improve the situation.

Sorry, I didn't mean to imply there was a bug. I just meant that
package-initialize is slow when you have lots and lots of packages,
simply because a lot of work is being done. The problem is that
package-initialize is a black box, so the user doesn't know where to
start if they want to optimize their init. It'd be helpful to know if
the bottleneck was:

* evaluating autoloads
* processing dependencies
* loading caches
* activating packages that are no longer referenced in the init-file ??
* etc

I know at least one person who was frustrated that a quarter of their
init-file was a black-box package-initialize call (at least before
they switched to straight.el).

> The source code of package.el is not supposed to be opaque. I wasn't
> the original author and I didn't find it opaque when I looked at it.
> So if it's opaque, maybe it's my fault, but I'm probably not the
> right guy to fix it, then ;-)

When I look at package.el, I see things like:

* not all the functions and variables have docstrings
* quite a few FIXME notices
* a method called "package--make-autoloads-and-stuff" (???)
* a lack of high-level explanation as to how things are organized and
  what the various formats are

Maybe if I worked with package.el more, I would not be as confused.
And maybe I have unrealistic expectations. But I think it would really
help if there were a comprehensive explanation of exactly how
package.el thinks about package installation and loading. For
reference, [3] is the corresponding explanation for straight.el.
Things like the meaning of "package", "installation", "deletion",
"version", "activate", etc. most likely seem obvious to everyone
working on package.el, but in fact there are many different ways to
handle these things (they are done very differently in straight.el
than in package.el, for example), so documentation is really
necessary.

[1]: 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Multi_002dfile-Packages.html
[2]: 
https://www.gnu.org/software/emacs/manual/html_node/elisp/Package-Archives.html#Package-Archives
[3]: https://github.com/raxod502/straight.el#conceptual-overview