Re: [PATCH v2] meson: Pass -j option to sphinx

[Fabiano Rosas]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote:
>> Save a bit of build time by passing the number of jobs option to
>> sphinx.
>> 
>> We cannot use the -j option from make because meson does not support
>> setting build time parameters for custom targets. Use nproc instead or
>> the equivalent sphinx option "-j auto", if that is available.
>> 
>> Also make sure our plugins support parallelism and report it properly
>> to sphinx. Particularly, implement the merge_domaindata method in
>> DBusDomain that is used to merge in data from other subprocesses.
>> 
>> before:
>>   $ time make man html
>>   ...
>>   [1/2] Generating docs/QEMU manual with a custom command
>>   [2/2] Generating docs/QEMU man pages with a custom command
>> 
>>   real    0m43.157s
>>   user    0m42.642s
>>   sys     0m0.576s
>> 
>> after:
>>   $ time make man html
>>   ...
>>   [1/2] Generating docs/QEMU manual with a custom command
>>   [2/2] Generating docs/QEMU man pages with a custom command
>> 
>>   real    0m25.014s
>>   user    0m51.288s
>>   sys     0m2.085s
>
> On my 12 CPU laptop I see a similar magnitude benefit - about
> 20 seconds is cut from the docs build time - 50 down to 30 secs.
>
> Watching the CPU usage I see sphinx is not very good at keeping
> all CPUs busy. For perhaps 2 seconds I'll see  8 sphinx processes
> burning CPUs, but the majority of the time it'll only be 1 or 2
> sphinx processes.
>
> IOW, we do get a benefit, but it is not nearly as good as one
> might hope for given the number of CPUs potentially available.
>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>>  docs/meson.build           | 12 ++++++++++++
>>  docs/sphinx/dbusdomain.py  |  4 ++++
>>  docs/sphinx/fakedbusdoc.py |  5 +++++
>>  docs/sphinx/qmp_lexer.py   |  5 +++++
>>  4 files changed, 26 insertions(+)
>
> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>
>> 
>> diff --git a/docs/meson.build b/docs/meson.build
>> index f220800e3e..9e4bed6fa0 100644
>> --- a/docs/meson.build
>> +++ b/docs/meson.build
>> @@ -10,6 +10,18 @@ if sphinx_build.found()
>>      SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
>>    endif
>>  
>> +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
>> +                               check: false).stdout().split()[1]
>> +  if sphinx_version.version_compare('>=5.1.2')
>> +    SPHINX_ARGS += ['-j', 'auto']
>> +  else
>> +    nproc = find_program('nproc')
>> +    if nproc.found()
>> +      jobs = run_command(nproc, check:false).stdout()
>> +      SPHINX_ARGS += ['-j', jobs]
>> +    endif
>> +  endif
>
> ANy reason for check: false in these 2 run_command calls ?
>

No, I haven't thought about it. I'll change them to true.