Re: [Qemu-devel] Sketch of a transition of QEMU docs to Sphinx

[John Snow]

On 5/21/19 2:56 PM, Peter Maydell wrote:
> Currently we have a vague plan that we should migrate our
> documentation away from Texinfo to using Sphinx, plus some isolated
> bits of documentation already in .rst format. This email is an attempt
> to sketch out a transition plan for getting us from where we are today
> to where (I think) we want to be.
> 
> 
> WHERE WE ARE TODAY
> ------------------
> 
> I'm going to concentrate on the documentation that's installed by
> 'make install', because anything else is for developers only, making
> it lower priority to clean up and more amenable to messing around with
> anyway.
> 

To an extent, I'd like to emphasize that our "users" are increasingly
developers from other projects and these internal docs are primarily
useful for this crowd.

> Currently we install:
> 
> - in $DESTDIR/usr/local/share/doc/qemu:
>  qemu-doc.html
>  qemu-ga-ref.html
>  qemu-qmp-ref.html
>  qemu-doc.txt
>  qemu-ga-ref.txt
>  qemu-qmp-ref.txt
>  interop/  (the only Sphinx manual we currently ship)
> 

Oh, but this is exactly what I was referencing above, so nevermind.

> - in $DESTDIR/usr/local/share/man/:
>  man1/qemu.1
>  man1/qemu-img.1
>  man1/virtfs-proxy-helper.1
>  man1/qemu-trace-stap.1
>  man8/qemu-ga.8
>  man8/qemu-nbd.8
>  man7/qemu-block-drivers.7
>  man7/qemu-cpu-models.7
>  man7/qemu-ga-ref.7
>  man7/qemu-qmp-ref.7
> (possibly I missed one or two files, because what we install
> depends on configure options and maybe my test build-n-install
> didn't build everything we might ship.)
> 
> 
> This documentation is generated from a mix of:
> 
> - hand-written texinfo:
>  qemu-doc.texi (a top level file with both content and @include directives)
>  qemu-deprecated.texi
>  qemu-ga.texi
>  qemu-img.texi
>  qemu-nbd.texi
>  qemu-option-trace.texi
>  qemu-tech.texi
>  docs/security.texi
>  docs/qemu-cpu-models.texi
>  docs/qemu-block-drivers.texi
>  docs/interop/qemu-ga-ref.texi
>  docs/interop/qemu-qmp-ref.texi
>  fsdev/virtfs-proxy-helper.texi
>  scripts/qemu-trace-stap.texi
> - texinfo sections extracted from *.hx files by hxtool
> - texinfo autogenerated from json by qapi-gen.py
> 
> (The .html and .txt files are generated from the texinfo directly; the
> manpages via texi2pod and pod2man, which effectively create the
> manpages from marked-up subsqections of the input texi.)
> 
> WHERE WE WANT TO BE
> -------------------
> 
> (This is based on the manual split outlined at the top of
> https://wiki.qemu.org/Features/Documentation which I am assuming we
> have general consensus for. NB that IMHO everything else on that
> wiki page except the first part explaining the manual structure
> is now out of date.)
> 
> - in $DESTDIR/usr/local/share/doc/qemu:
>  interop/ sphinx manual
>  system/ sphinx manual (system emulation user's guide)
>  user/ sphinx manual (user-mode emulation user's guide)
>  specs/ sphinx manual (guest hardware etc specs)
>  plus maybe text versions of these (interop.txt, system.txt, etc)
> 
> - in $DESTDIR/usr/local/share/man/:
>  the same set of manpages we have currently
> 
> - not actually installed:
>  devel/ sphinx manual
> 
> Sphinx supports a "plain text" output format, which will create a
> one-big-text-file for each of the four installed manuals. It might
> also be possible to generate some kind of "one .txt file per input
> .rst file" format, but that would require a greater amount of messing
> about.  (The sphinx text builder doesn't do this and so you'd need to
> get make to invoke sphinx once per rst file in each manual, which will
> be awkward.) Or we could just say "it's 2019, the HTML documentation
> is the official format, that (and manpages) is what we ship". Opinions?
> 

I am not sure what the value in plaintext Sphinx docs are, since they
lose the hyperlinks ... unless you keep it in ReST or Markdown format,
and then you could just install the source documents, no?

I just seem to think that most of the text-only purists are also the
type to already have the source tree checked out somewhere.

As long as we've got HTML + manpages, I think we're doing OK.

> Sphinx supports a "manpage" builder, which I have not yet investigated
> closely but which I'm hoping will do what we need. It works by
> specifying a list of "this .rst becomes this manpage in this
> manpage-section", so we can for instance have the qemu-ga-ref and
> qemu-qmp-ref be subsections of the HTML interop/ manual but also
> pulled out into their own manpages. (It is also possible to make
> sections of a .rst file be output only for particular builders, but
> there seem to be some limitations on it -- notably it filters the
> output but it doesn't affect things like tables of contents for
> formats which have them. I am hoping we can do what we want without it.)
> 
> If there are manpages we currently ship which aren't actually very
> useful and which we could just drop that would be useful to know.
> 
> HOW TO GET THERE FROM HERE
> --------------------------
> 
> Some parts of this are easier than others (in an "everything depending
> on everything else" sense; conversion of documents and writing new
> generate-rst-file backends is obviously a fair amount of work).
> 
> (0) This is all independent of Gabriel's work on generating
> API documentation, because that will just go into the devel/
> manual, and doesn't interact at all with the user-facing manuals.
> 
> (1) qemu-ga-ref and qemu-qmp-ref will become part of the interop/
> manual. This requires:
>  * adding rst-generation support to qapi-gen.py
>  * converting the hand-written texinfo parts to rst
>  * listing those new .rst files in docs/interop/index.rst
>  * makefile runes to create manpages via the 'man' sphinx builder
>    from docs/interop/qemu-ga-ref.rst and qemu-qmp-ref.rst
>  * there is a tests/qapi-schema/doc-good.test.texi which I think
>    is acting as a test of the texi doc generation; this should
>    be converted to test the rst doc generation instead
>  * delete all the old texinfo build machinery, install runes, etc
> 

Good riddance. I assume this also means we need to rewrite the QAPI
comments to use Sphinx-ese instead of Texi-ese.

I think that'd be good, though, because it means we get proper lexing
and parsing on our examples. Especially in places where we
cross-reference; for example -- QMP has this trick where our commands
don't take any arguments themselves but rather define a data type it
expects as input.

In the output manual, this means that we don't document the arguments
near the function, which is awful.

Sphinx can help us cross-reference these things better.

> We could do this as its own patch series which would end up with a
> result that would be fine to ship, even if we don't get any of the
> other manuals converted for a particular QEMU release.  It's the
> easiest part because these docs already have an obvious home in a
> Sphinx manual we're already shipping.
> 
> (2) Any parts of the texi docs which are really "developer docs"
> should be moved into devel/
> 
> This is easy because we don't need to maintain a very coherent manual
> for devel/, we can just have it be a bucket of random documents for
> the moment. In particular:
> 
>  * the qemu-tech.texi "translator internals" section should be moved
>    (I suspect it's a bit out of date so we should add a warning note
>    at the top of the new file about that)
>  * I think we should just delete the "QEMU compared to other emulators"
>    section of qemu-tech.texi (and the accompanying bibliography of URLs)
>    since it hasn't been updated in several years and it's not really
>    documenting QEMU. This sort of general-information stuff would
>    live better on the wiki if we care about it at all.
>  * I don't think there's anything else that's really developer
>    documentation, but I haven't looked too closely
> 
> We could do this now if we wanted, independently of (1).
> 
> (3) Things which are mostly standalone manpages
> 
> Round about here things start to get trickier because we are looking
> at things which are part of the big fat qemu-doc.html.  The easiest
> next pieces to pull out are the ones which are standalone manpages
> which we're also just using @include to stuff into qemu-doc.html:
> 
>  * qemu-ga.texi
>  * qemu-nbd.texi
>  * qemu-img.texi
>  * qemu-cpu-models.texi
>  * qemu-block-drivers.texi
>  * virtfs-proxy-helper.texi
>  * scripts/qemu-trace-stap.texi
> 
> We can:
>  * convert each of these to a .rst file in the right sphinx
>    manual (probably mostly system/)
>  * generate the manpage output using sphinx as we did for qemu-ga-ref etc
>  * just drop the @include from qemu-doc.texi
> 
> qemu-img.texi here will be the tricky one as it uses hxtool to
> generate the texi for the command line option descriptions. I'm
> not sure how best to handle hxtool, because it is really just
> a mechanism for slicing the right fragments out of the input file.
> Perhaps we need to add SRST/ERST directives to match the STEXI/ETEXI
> ones we have at the moment.
> 
> This will result in a slightly "bucket of distinct docs" feel to the
> system/ manual at this point, but these files are all basically mostly
> self-contained, so it wouldn't be too terrible to ship a QEMU in this
> state.
> 
> (4) Everything else
> 
> At this point what we have left is:
>  * qemu-doc.texi itself
>  * qemu-deprecated.texi
>  * docs/security.texi
>  * qemu-option-trace.texi
>  * the leftovers from qemu-tech.texi
>  * the stuff generated via hxtool from hmp-commands-info.hx,
>    hmp-commands.hx and qemu-options.hx
> 
> and we pretty much need to:
>  * convert all that over to rst in the system/ and user/ manuals
>    (perhaps sometimes interop/)
>  * generate the qemu.1 manpage
>  * delete all the leftover machinery
> 
> We could do at least some of this in chunks, but if we had to release
> QEMU midway through this part of the transition we'd have a very weird
> setup where half the core-emulator-docs were in the old html file and
> the other half were in the new style manuals, so we should definitely
> try to avoid doing that.
> 
> PROBLEMS
> --------
> 
> I haven't yet thought through what exactly might turn out
> to be nasty problems with the conversion, and it's getting
> late in the day here, so I leave that part to you :-)
> 
> NB: Sphinx does also have a texinfo output option, but I have
> not used it in this transition, because I suspect it will want
> to output an entire texinfo document rather than a fragment that
> could be included in a larger document, which makes it a bit less
> useful to us. If we think step (4) above is too big-bang then
> we could investigate whether it would be workable to convert
> some files to .rst but then generate .texi from them to include
> in qemu-doc.texi until we're ready to flip everything over to
> directly building html.
> 
> Comments on this whole proposal (whether I've forgotten anything
> about our current setup, whether we really do want to go to the
> place I've suggested we go, etc) welcome.
> 
> thanks
> -- PMM
> 

It looks good to me overall -- any progress towards a unified manual
with automatic checking of any sort is an improvement.

I'll try to keep an eye on it over the summer and see if I can't help
nudge the block layer documents along the path to something unified.
I've written one custom extension now, so what's a few more?

I also wrote a hasty script that tried to unify the qemu-img.texi and
qemu-img help outputs and removed one more usage of the hxtool, but it's
only a stopgap and wasn't something like proper sphinx integration, but
maybe I could keep working in that area soon.

I would also like to say that it would be useful to host our HTML docs
online such that we can point to the different versions, like you can
with the official python docs. They often have a little dropdown where
you can switch between 3.5/3.6/3.7 or so on and see the difference on
that exact page. It'd just simply be nice to be able to do the same for
QMP docs.

--js