Re: Allow parallelism in input/regression/lilypond-book/ (issue 547680043 by address@hidden)

[David Kastrup]

Dan Eble <address@hidden> writes:

> On Feb 23, 2020, at 09:11, David Kastrup <address@hidden> wrote:
>> 
>>>  "Sharing Job Slots with GNU make"
>>>  https://www.gnu.org/software/make/manual/html_node/Job-Slots.html
>> 
>> But that still doesn't solve the problem that the database approach of
>> lilypond-book does not work for running multiple lilypond-book jobs in
>> parallel.
>
> Maybe I haven't devoted enough time to understand this situation well.
> Basically, my thought was that whatever currently uses CPU_COUNT could
> communicate with the job server instead so that one wouldn't need to
> repeat -jN CPU_COUNT=N.

It is LilyPond itself that gets started with a -djob-count argument and
a long commandline of files which it then distributes to the named jobs.

make/lilypond-vars.make:LILYPOND_JOBS=$(if 
$(CPU_COUNT),-djob-count=$(CPU_COUNT),)

dak@lola:/usr/local/tmp/lilypond$ git grep LILYPOND_JOBS
make/lilypond-vars.make:LILYPOND_JOBS=$(if 
$(CPU_COUNT),-djob-count=$(CPU_COUNT),)
make/lilypond-vars.make:$(LILYPOND_JOBS) \
make/lysdoc-targets.make:       $(MAKE) 
LILYPOND_BOOK_LILYPOND_FLAGS="-dbackend=eps --formats=ps $(LILYPOND_JOBS) 
-dseparate-log-files -dinclude-eps-fonts -dgs-load-fonts --header=texidoc -I 
$(top-src-dir)/Documentation/included/ -ddump-profile -dcheck-internal-types 
-ddump-signatures -danti-alias-factor=1 
-dfont-export-dir=$(top-build-dir)/out-fonts -O TeX-GS" LILYPOND_BOOK_WARN= 
$(outdir)/collated-files.html LYS_OUTPUT_DIR=$(top-build-dir)/out/lybook-testdb
make/lysdoc-targets.make:       $(MAKE) 
LILYPOND_BOOK_LILYPOND_FLAGS="-dbackend=eps --formats=ps $(LILYPOND_JOBS) 
-dseparate-log-files -dinclude-eps-fonts -dgs-load-lily-fonts --header=texidoc 
-I $(top-src-dir)/Documentation/included/ -ddump-profile -dcheck-internal-types 
-ddump-signatures -danti-alias-factor=1" LILYPOND_BOOK_WARN= 
$(outdir)/collated-files.html LYS_OUTPUT_DIR=$(top-build-dir)/out/lybook-testdb
scripts/auxiliar/build-coverage.sh:  make conf=cov test-clean OUT_TEST=testcov 
LILYPOND_JOBS= && \
scripts/auxiliar/build-coverage.sh:  make conf=cov test OUT_TEST=testcov 
LILYPOND_JOBS='-dtrace-scheme-coverage '

While we could try asking parallel Make for job slots, the problem
remains that only one lilypond-book job (per backend/database) could
possibly run at a time.

We could, however, conceivably parallelize a lilypond-book job with PNG
backend and a lilypond-book job with PDF backend.  I don't think that
those would share the same database (correct me if I am wrong).  I have
no good idea how to make this parallelisation work in anything remotely
resembling a reproducible/reliable manner, however.  At the time the
first lilypond-book job is started, it needs to get told how many
processors it should be using, and by that time the job server has no
idea how many other lilypond-book jobs may appear in parallel.

-- 
David Kastrup
My replies have a tendency to cause friction.  To help mitigating
damage, feel free to forward problematic posts to me adding a subject
like "timeout 1d" (for a suggested timeout of 1 day) or "offensive".