Re: parallelization of ./configure compiler test processes

[Bob Friesenhahn]

On Thu, 30 Mar 2023, Thomas Jahns wrote:

> Hello Danny,
>
> I spent some time thinking about improvements to autoconf configure scripts 
> (while waiting for builds to proceed). In my view, it is currently still easier 
> to seek small efficiency gains that, in sum, could still improve run-time 
> substantially than parallelizing the whole would be, because there is so much 
> often untapped potential:
>
> * Use bash builtin shell commands to fork and especially exec less. In modern 
> systems with comparatively fast data paths compared to anything that affects 
> resource control, like changing memory mappings, dropping caches etc., syscalls 
> can be a substantial source of slow down.
> * Use TMPDIR to prevent temporary files from hitting disk (use /dev/shm or 
> similar instead of /tmp)
> * In the case of spack I've seen substantial improvements from running the 
> whole build in /dev/shm and think spack should at least mention the possibility 
> for systems with substantial amounts of RAM (and guess what kind of system many 
> sites using spack just happen to have?).
> * The gcc option -pipe is similarly helpful to get build phases to start as 
> soon as possible.

The main problem with the above is that it is focused on one 
particular shell, and operating system.  While the combination is very 
popular, Autoconf is still also a portability tool.

While continued optimization is important, I suggest that a good 
strategy is to also spend more time on investigating useful caching 
features.  It is common that a given test permutation is executed more 
than once.  By implementing a smart shared cache, it should be 
possible to identify similar permutations and cache the results so 
that the next time they are encountered, the cached results will 
simply be used.  The cache could be at the user and/or whole system 
level.

Autoconf already has good support for a local cache so that a 
re-configure is much faster, and it has support for a system-wide 
shared cache.

Speed-up due to caching can be explored by using the --config-cache 
option, or using a config.cache file.

One would think that a "compiler" test should be cacheable given the 
same compiler with similar options.

Bob
--
Bob Friesenhahn
bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
Public Key,     http://www.simplesystems.org/users/bfriesen/public-key.txt