Re: Managing environments (Python venv, guix environment, etc.)

[Eli Zaretskii]

> From: address@hidden
> Date: Mon, 25 Jul 2016 00:50:34 -0400
> 
> > IOW, I simply fail to see how we will be able to avoid disrupting the
> > most basic features if we modify exec-path.  Even visiting a file can
> > fire up a subprocess -- how do we make sure the right program for that
> > will still be found, if we let some project's environment mess with
> > exec-path behind the user's back?  And what about spellers (e.g., for
> > ispell-check-comments-and-strings)?  Etc. etc.
> 
> I don't think that would be an issue. At the moment, the primary place
> to use these kind of special environments is a Unix shell. If basic
> tools like coreutils weren't in the environment, the environment would
> already be useless for the user. And the only way to get some kind of
> useless environment like that would be to explicitly configure it; I'm
> inclined to call that user error. Also: These special environments
> typically just prepend new directories to PATH and other vars,
> "inheriting" the binaries that were in the user's pre-existing
> environment, and just overriding some, so in the typical case things
> will certainly work fine.
> 
> Also note: If a binary from outside some project-specific environment is
> run within that environment, it might break. (Maybe you preload some
> crazy library within the project-specific environment, and have a set of
> project-specific binaries that know how to cope with that.)

See, the above 2 paragraphs contradict each other.  The second one is
precisely the kind of breakage I had in mind and feared of, and I see
now that we are in violent agreement regarding such a possibility.

Prepending directories to PATH doesn't solve these problems.  E.g., if
the prepended directory holds a version of gzip that is "broken" in
the above-mentioned meaning in the environment of a project, then
Emacs will be unable to visit gzip-compressed files, such as Info
manuals and its own Lisp sources.

I think we need to avoid this danger.

> So if we don't modify exec-path to be project-specific, then anything
> that wants to run a process inside the project-specific environment
> would need to be explicitly modified to search the project-specific
> path, to avoid running outside binaries inside the special
> environment.

I think the suggestion to define a series of variables, which, if
non-nil, will override the exec-path search by the relevant
primitives, avoids this issue, because these variables will be handled
on the infrastructure level.

> > Once again, why not use locate-file?  CEDET and EDE already do, and I
> > assume for good reasons.
> 
> Yes, a new program that wants to search per-environment search paths
> could use locate-file. But I still hope for not needing to modify things
> to be environment-aware.

We cannot avoid some modifications anyway.  You are talking about
changing Emacs on a very low and delicate level.

> > More generally, why would a project want to modify exec-path? to find
> > which programs?  Is it only the compiler/interpreter of the
> > programming language used by the project, or something else as well,
> > and why?  E.g., I don't really understand your example with browse-url
> > -- why would I want to change a browser as function of the project I'm
> > working on?
> [...]
> maybe if you're developing a browser, you'd want browse-url to pick
> up the version compiled in the project-specific environment when
> you're working on it.

browse-url already has browse-url-*-program and
browse-url-default-browser variables that should be enough to cater to
this use case.  A project can simply customize their values, and make
them buffer-local if needed.

> In general I think it's entirely reasonable to want to be able to run
> project-specific version of developer tools. The compiler/interpreter is
> the most obvious project-specific program, but there are benefits to
> supporting running every developer tool inside the project-specific
> environment.

But exec-path is not just for development tools.  It's for _any_
program Emacs might need to run.  Thus changing it has an effect that
is much more wide and prominent.

> Here are three benefits that I see for running everything within the
> project-specific environment by default:
> - This works identically (and therefore compatibly) to the Unix shell
> inside a special environment.

Actually, many development environments are configured by pointing at
specific absolute file names of the tools, such that any changes in
PATH don't affect them.

> - Any other project-specific developer tools that we don't know about or
> don't yet exist, will be automatically supported.
> - And most importantly: all the myriad programming modes for Emacs with
> their many custom methods for running their compiler/interpreter, will
> automatically support running their compilers/interpreters in an
> environment-aware way.

I think the downsides of this are too serious to ignore.  Like this
one, for example:

> However I admit that if the user wants to run a development program from
> outside the project-specific environment (or even a development program
> from another environment) it could become a little more tricky...

More generally, running _any_ program that doesn't belong to the
environment might subtly break.

> And the conceptually easiest way is still just updating Elisp commands
> (such as compile or shell-command) one by one to be environment-aware.

The challenge is to make changes to the infrastructure that cover the
80% of use cases in environment-aware development, without making the
cure worse than the disease.