Re: Parallelization of shell scripts for 'configure' etc.

[Zack Weinberg]

On Thu, Jun 16, 2022, at 12:13 AM, Warren Young wrote:
> A Makefile can be viewed as a series of shell scripts interconnected by
> dependency rules that dictate which elements can be safely skipped, which
> must be run, and in what order.
>
> I believe it's possible to recast Autoconf so that it generates a Makefile
> that produces the same outputs as a configure script, but which can be run
> under "make -jN" to run in parallel, safely.

Probably, yeah.  I like the idea of using a Makefile, that means we can 
outsource the "do up to N things in parallel" part to the Make implementation.  
I have about half of a patch on my local HDD that converts auto*test* to that 
style of parallelism, and if I ever get the time to finish it, it might be 
adaptable to autoconf.

Potential lower-hanging fruit, though, is to add hand-coded parallelism inside 
AC_CHECK_{DECLS,FILES,HEADERS,FUNCS,MEMBERS,PROGS,TOOLS,TYPES}.  I wouldn't be 
surprised if many existing configure scripts spend the bulk of their time 
inside these, and they are already looping over lists, where we know the answer 
to each iteration is independent of all the others.

Whatever we do needs to be carefully designed to avoid introducing 
nondeterminism in the contents of config.status and everything downstream of 
it, and each check's log output should be appended atomically to config.log.

zw