Re: native compilation units

[Stefan Monnier]

>> Performance issues with read access to directories containing less than
>> 10K files seems like something that was solved last century, so
>> I wouldn't worry very much about it.
> Per my response to Eli, I see (network) directories become almost unusable
> somewhere around 1000 files,

I don't doubt there are still (in the current century) cases where
largish directories get slow, but what I meant is that it's now
considered as a problem that should be solved by making those
directories fast rather than by avoiding making them so large.

>> [ But that doesn't mean we shouldn't try to compile several ELisp files
>>   into a single ELN file, especially since the size of ELN files seems
>>   to be proportionally larger for small ELisp files than for large
>>   ones.  ]
>
> Since I learned of the native compiler in 28.1, I decided to try it out and
> also "throw the spaghetti at the wall" with a bunch of packages that
> provide features similar to those found in more "modern" IDEs.  In terms of
> startup time, the normal package system does not deal well with hundreds of
> directories on the load path, regardless of AOR native compilation, so I'm
> tranforming the packages to install in the version-specific load path, and
> compiling that ahead of time.  At least for the ones amenable to such
> treatment.

There are two load-paths at play (`load-path` and
`native-comp-eln-load-path`) and I'm not sure which one you're taking
about.  OT1H `native-comp-eln-load-path` should not grow with the number
of packages so it typically contains exactly 2 entries, and definitely
not hundreds.  OTOH `load-path` is unrelated to native compilation.

I also don't understand what you mean by "version-specific load path".

Also, what kind of startup time are you talking about?
E.g., are you using `package-quickstart`?

> Given I'm compiling all the files AOT for use in a common installation
> (this is on Linux, not Windows), the natural question for me is whether
> larger compilation units would be more efficient, particularly at startup.

It all depends where the slowdown comes from :-)

E.g. `package-quickstart` follows a similar idea to the one you propose
by collecting all the `<pkg>-autoloads.el` into one bug file, which
saves us from having to load separately all those little files.  It also
saves us from having to look for them through those hundreds
of directories.

I suspect a long `load-path` can itself be a source of slow down
especially during startup, but I haven't bumped into that yet.
There are ways we could speed it up, if needed:

- create "meta packages" (or just one containing all your packages),
  which would bring together in a single directory the files of several
  packages (and presumably also bring together their
  `<pkg>-autoloads.el` into a larger combined one).  Under GNU/Linux we
  could have this metapackage be made of symlinks, making it fairly
  efficient an non-obtrusive (e.g. `C-h o` could still get you to the
  actual file rather than its metapackage-copy).
- Manage a cache of where are our ELisp files (i.e. a hash table
  mapping relative ELisp file names to the absolute file name returned
  by looking for them in `load-path`).  This way we can usually avoid
  scanning those hundred directories to find the .elc file we need, and
  go straight to it.

> I posed the question to the list mostly to see if the approach (or similar)
> had already been tested for viability or effectiveness, so I can avoid
> unnecessary experimentation if the answer is already well-understood.

I don't think it has been tried, no.

> I don't know enough about modern library loading to know whether you'd
> expect N distinct but interdependent dynamic libraries to be loaded in as
> compact a memory region as a single dynamic library formed from the same
> underlying object code.

I think you're right here, but I'd expect the effect to be fairly small
except when the .elc/.eln files are themselves small.

> It's not clear to me whether those points are limited to call
> sites or not.

I believe it is: the optimization is to replace a call via `Ffuncall` to
a "symbol" (which looks up the value stored in the `symbol-function`
cell), with a direct call to the actual C function contained in the
"subr" object itself (expected to be) contained in the
`symbol-function` cell.

Andrea would know if there are other semantic-non-preserving
optimizations in the level 3 of the optimizations, but IIUC this is very
much the main one.

>> IIUC the current native-compiler will actually leave those
>> locally-defined functions in their byte-code form :-(
> That's not what I understood from
> https://akrl.sdf.org/gccemacs.html#org0f21a5b
> As you deduce below, I come from a Scheme background - cl-flet is the form
> I should have referenced, not let.

Indeed you're right that those functions can be native compiled, tho only if
they're closed (i.e. if they don't refer to surrounding lexical
variables).
[ I always forget that little detail :-(  ]

> It seems more straightforward than trying to link the eln
> files into larger units after compilation.

That seems like too much trouble, indeed.


        Stefan