Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> Rather, as a user I don't want to have to prescribe whether
> package.el should be enabled or disabled, by default or
> otherwise.  I don't want to have to care about package.el.
>
> I would like accessing packages to just happen, when/if I
> ask to access them - e.g. if I do `M-x list-packages'.

I think this is how most users feel. And similarly to you, most users
aren't intimately familiar with the technical challenges of
integrating a package manager into Emacs. So I thought it might be
nice to briefly outline the implications of this goal.

[Well, I guess it's only brief by *my* standards. Sorry...]

1. Accessing most package manager operations on-demand is trivial. M-x
   package-list-packages is autoloaded, and information about
   available packages can be loaded when the function is invoked. This
   is not being debated.

2. Having your packages automatically be enabled is more complicated.
   This is where package.el diverges from other packages. For example,
   in the case of bookmark.el, your bookmarks are not loaded until you
   try to access bookmarks. In the case of packages which provide a
   passive effect, like electric-pair-mode, they will only be enabled
   automatically if you put some code to that effect in your
   init-file.

3. On the other hand, we want your installed packages to be available
   automatically without such code being present (or at least most
   people do; I personally don't). Practically, this means that
   `package-initialize' has to get called at some point during
   initialization; there is simply no other way for packages to be
   enabled.

4. We therefore have two options: require the user to call
   `package-initialize' in their init-file, or call it in startup.el.
   The former is more modular and I would prefer it, but if we work
   under the assumption that under all circumstances we want installed
   packages to be activated automatically, then we have to call
   `package-initialize' in startup.el.

5. It's a non-starter to call `package-initialize' before the
   init-file is loaded, since then it is impossible to customize
   package.el (unless you introduce a second init-file which is loaded
   even earlier). Thus `package-initialize' has to be called after
   loading the init-file.

6. This satisfies all the conditions you asked for: namely, that you
   don't have to care about package.el. However, there is one final
   problem: you will probably want to put some Lisp code in your
   init-file to customize your installed packages, and this can only
   be done after `package-initialize' is called.

7. The only way to handle this is, therefore, to call
   `package-initialize' in your init-file if you wish to customize
   your installed packages directly in the init-file. As a bookkeeping
   measure, of course, if `package-initialize' is called in the
   init-file, then it's not called again after loading the init-file.

8. Thus we are led inevitably to the conclusion that the natural state
   for the user's init-file is that it has `package-initialize' in it
   somewhere (i.e. after package.el configuration, if any, but before
   package configuration, if any). The question is how to make it easy
   for users to realize that this is the case.

9. A bad solution to this problem is to have Emacs write a
   `package-initialize' call into the init-file. This is bad because
   the call needs to go in a specific place, and possibly in a
   different file that is loaded from the init-file, and so the
   insertion location will usually be incorrect. It's also bad because
   Emacs should keep its hands off the init-file, as you've pointed
   out.

10. A better solution, though not ideal, is to have Emacs generate a
    template init-file with a `package-initialize' call if no
    init-file exists. That way, when users go to add Lisp code to
    customize their packages, they will see a comment saying "make
    sure to place Lisp code for customizing packages *after* this
    `package-initialize' call", and they will therefore do the right
    thing. It's been well-established that such "just-in-time nudges"
    are very effective at getting users to take the correct action.
    Much more effective than external documentation, for example.

11. Alternatively, we can just say that if you are writing Lisp code
    to customize your packages, it's your responsibility to initialize
    the package manager manually. I'd like to move to this point
    eventually, but for now I think the template init-file is a good
    middle ground.

> If by Emacs going through contortions you mean that it does
> some work under the covers then no, I don't care whether it
> does that.

That's reasonable. So you are probably fine with `package-initialize'
being called in startup.el.

> I've already said that I don't think Emacs should write to
> init files.  (And I don't think it should have to do that.
> Why is that considered a hard requirement?)

I don't mean that Emacs writing to init-files is a requirement. Like
you, I don't think Emacs should do that. What I meant was that having
a call to `package-initialize' in the init-file is a hard requirement
for being able to put customization of installed packages into the
init-file.

> But I've also said that if the package system needs to
> record some info persistently for a user then it should use
> a different file from the init file.  That doesn't mean that
> I disagree with Emacs writing such info locally.  It just
> means hands-off init files, please.

Yeah. The problem is that we're not talking about the persistent info,
which is already stored in ~/.emacs.d/elpa. We're talking about the
code that initializes package.el. This can be compared with the code
that enables a minor mode: it has to be in the user's init-file if
it's going to take effect. The issue is how to encourage users to set
up their init-file correctly to use package.el without having Emacs
modify the init-file.

> If it can do what it needs to do by writing to an init file then why
> can't it do what it needs to do by using a separate file?

To reiterate, that would just move the problem, since
`package-initialize' is, in essence, the function that loads this
separate file.

> Is the problem the difficulty for a user to determine where in the
> init file to load the separate file?

No, the problem is that users forget to put this loading call in the
init-file at all.

> If so, then Emacs can perhaps detect bad placement - not by
> examining the init file but by the effects of this or that
> inappropriate placement. E.g., if it is loaded before XYZ has been
> defined or initialized then tell the user to move it etc.

That's a wonderful idea, and I think that regardless of any results of
my proposal, we should implement this error checking as well.

> Debatable whether it is less bothersome/intrusive for a user.

Well, I suppose it is indeed debatable. But you can't argue that the
current behavior is more predictable than the proposed behavior, at
least, and certainly not that the current behavior is less dangerous.

We could eliminate the annoyance for existing users by allowing to
create a dotfile in ~/.emacs.d that tells Emacs not to create a
template init-file, even if none exists already. We could go even
farther and have the template init-file create that dotfile, so the
template init-file would really be a one-time thing.

I know I argued against magic dotfiles earlier, but I think it's
different in this case: the dotfile would now be customizing a part of
the startup process that is fundamentally impossible to put into the
init-file, rather than a single tiny part of Emacs' business logic,
i.e. one of its package managers.

> I'll make the argument.

And I'll join you just as soon as this proposal goes through.

> lots of users (especially nowadays) do not read doc. Create a good
> video presenting the same relevant information and that might be all
> that's needed.

True. I think the template init-file would help with this by putting
the relevant information in the right place at the right time, in a
hard-to-ignore way that is nevertheless not annoying like a popup
would be.

> Create a good video presenting the same relevant information and
> that might be all that's needed.

Like, literally a video? Or did you mean an interactive Emacs-y
tutorial thing? If the former, I worry it might be difficult for
people to find.

> That plus making the package system perform some checks to
> recognize problematic situations resulting from inappropriate
> use - and then point to relevant videos/doc etc.

+1