> There was a thread in January starting at
> https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01005.html that
> gets at one scenario. At least in pre-10 versions in my experience,
> Windows has not dealt well with large numbers of files in a single
> directory, at least if it's on a network drive.
Hmm... I count a bit over 6K ELisp files in Emacs + (Non)GNU ELPA, so
the ELN cache should presumably not go much past 10K files.
Performance issues with read access to directories containing less than
10K files seems like something that was solved last century, so
I wouldn't worry very much about it.
Per my response to Eli, I see (network) directories become almost unusable somewhere around 1000 files, but it seems that's a consequence of the network and/or security configuration.
[ But that doesn't mean we shouldn't try to compile several ELisp files
into a single ELN file, especially since the size of ELN files seems
to be proportionally larger for small ELisp files than for large
ones. ]
Since I learned of the native compiler in 28.1, I decided to try it out and also "throw the spaghetti at the wall" with a bunch of packages that provide features similar to those found in more "modern" IDEs. In terms of startup time, the normal package system does not deal well with hundreds of directories on the load path, regardless of AOR native compilation, so I'm tranforming the packages to install in the version-specific load path, and compiling that ahead of time. At least for the ones amenable to such treatment.
Given I'm compiling all the files AOT for use in a common installation (this is on Linux, not Windows), the natural question for me is whether larger compilation units would be more efficient, particularly at startup. Would there be advantages comparable to including packages in the dump file, for example?
I posed the question to the list mostly to see if the approach (or similar) had already been tested for viability or effectiveness, so I can avoid unnecessary experimentation if the answer is already well-understood.
> Aside from explicit interprocedural optimization, is it possible libgccjit
> would lay out the code in a more optimal way in terms of memory locality?
Could be, but I doubt it because I don't think GCC gets enough info to
make such a decision. For lazily-compiled ELN files I could imagine
collecting some amount of profiling info to generate better code, but
our code generation is definitely not that sophisticated.
I don't know enough about modern library loading to know whether you'd expect N distinct but interdependent dynamic libraries to be loaded in as compact a memory region as a single dynamic library formed from the same underlying object code.
> If the only concern for semantic safety with -O3 is the redefinability of
> all symbols, that's already the case for emacs lisp primitives implemented
> in C.
Not really:
- Most ELisp primitives implemented in C can be redefined just fine.
The problem is about *calls* to those primitives, where the
redefinition may fail to apply to those calls that are made from C.
- While the problem is similar the scope is very different.
From Andrea's description, this would be the primary "unsafe" aspect of intraprocedural optimizations applied to one of these aggregated compilation units. That is, that the semantics of redefining function symbols would not apply to points in the code at which the compiler had made optimizations based on assuming the function definitions were constants. It's not clear to me whether those points are limited to call sites or not.
> It should be similar to putting the code into a let block with all
> defined functions bound in the block, then setting the global
> definitions to the locally defined versions, except for any variations
> in forms with semantics that depend on whether they appear at
> top-level or in a lexical scope.
IIUC the current native-compiler will actually leave those
locally-defined functions in their byte-code form :-(
That's not what I understood from
As you deduce below, I come from a Scheme background - cl-flet is the form I should have referenced, not let.
IOW, there are lower-hanging fruits to pick first.
This is mainly of interest if a simple transformation of the sort I originally suggested can provide benefits in either reducing startup time for large sets of preloaded packages, or by enabling additional optimizations. Primarily the former for me, but the latter would be interesting. It seems more straightforward than trying to link the eln files into larger units after compilation.
> It might be interesting to extend the language with a form that
> makes the unsafe optimizations safe with respect to the compilation unit.
Yes, in the context of Scheme I think this is called "sealing".
Stefan
No