Re: [Qemu-devel] Sketch of a transition of QEMU docs to Sphinx

[Peter Maydell]

On Tue, 28 May 2019 at 20:09, John Snow <address@hidden> wrote:
>
>
>
> On 5/21/19 2:56 PM, Peter Maydell wrote:
> > Currently we have a vague plan that we should migrate our
> > documentation away from Texinfo to using Sphinx, plus some isolated
> > bits of documentation already in .rst format. This email is an attempt
> > to sketch out a transition plan for getting us from where we are today
> > to where (I think) we want to be.
> >
> >
> > WHERE WE ARE TODAY
> > ------------------
> >
> > I'm going to concentrate on the documentation that's installed by
> > 'make install', because anything else is for developers only, making
> > it lower priority to clean up and more amenable to messing around with
> > anyway.
> >
>
> To an extent, I'd like to emphasize that our "users" are increasingly
> developers from other projects and these internal docs are primarily
> useful for this crowd.
>
> > Currently we install:
> >
> > - in $DESTDIR/usr/local/share/doc/qemu:
> >  qemu-doc.html
> >  qemu-ga-ref.html
> >  qemu-qmp-ref.html
> >  qemu-doc.txt
> >  qemu-ga-ref.txt
> >  qemu-qmp-ref.txt
> >  interop/  (the only Sphinx manual we currently ship)
> >
>
> Oh, but this is exactly what I was referencing above, so nevermind.

Right. The devel/ manual is for things which are purely
QEMU internals -- autogenerated internal API docs,
how TCG works, that kind of thing. (What we have so far
is in docs/devel/.) The interop/ manual is the one for
QEMU's interfaces to other projects.

> > Sphinx supports a "plain text" output format, which will create a
> > one-big-text-file for each of the four installed manuals. It might
> > also be possible to generate some kind of "one .txt file per input
> > .rst file" format, but that would require a greater amount of messing
> > about.  (The sphinx text builder doesn't do this and so you'd need to
> > get make to invoke sphinx once per rst file in each manual, which will
> > be awkward.) Or we could just say "it's 2019, the HTML documentation
> > is the official format, that (and manpages) is what we ship". Opinions?
> >
>
> I am not sure what the value in plaintext Sphinx docs are, since they
> lose the hyperlinks ... unless you keep it in ReST or Markdown format,
> and then you could just install the source documents, no?
>
> I just seem to think that most of the text-only purists are also the
> type to already have the source tree checked out somewhere.
>
> As long as we've got HTML + manpages, I think we're doing OK.

That tends to be my opinion (especially since we don't for instance
check the text output to make sure it hasn't rendered in some
way that introduces problems). I just wanted to give people
an opportunity to say "I like the way we currently ship plain text".

> > (1) qemu-ga-ref and qemu-qmp-ref will become part of the interop/
> > manual. This requires:
> >  * adding rst-generation support to qapi-gen.py
> >  * converting the hand-written texinfo parts to rst
> >  * listing those new .rst files in docs/interop/index.rst
> >  * makefile runes to create manpages via the 'man' sphinx builder
> >    from docs/interop/qemu-ga-ref.rst and qemu-qmp-ref.rst
> >  * there is a tests/qapi-schema/doc-good.test.texi which I think
> >    is acting as a test of the texi doc generation; this should
> >    be converted to test the rst doc generation instead
> >  * delete all the old texinfo build machinery, install runes, etc
> >
>
> Good riddance. I assume this also means we need to rewrite the QAPI
> comments to use Sphinx-ese instead of Texi-ese.

Yes, if we auto-inline comment text we would need to update
the markup (as with the .hx file stuff I mention later).
(I guess in theory if there's only a very limited set of
markups used we could make the rst generation translate from
the current format to rst, but I think we'd be better off
having the format really match our current documentation.)

> I think that'd be good, though, because it means we get proper lexing
> and parsing on our examples. Especially in places where we
> cross-reference; for example -- QMP has this trick where our commands
> don't take any arguments themselves but rather define a data type it
> expects as input.
>
> In the output manual, this means that we don't document the arguments
> near the function, which is awful.
>
> Sphinx can help us cross-reference these things better.

Yes, at least in theory. I haven't looked into how difficult
it is to get Sphinx to actually make things refer to each other,
but it really ought to be doable.


> > Comments on this whole proposal (whether I've forgotten anything
> > about our current setup, whether we really do want to go to the
> > place I've suggested we go, etc) welcome.

> It looks good to me overall -- any progress towards a unified manual
> with automatic checking of any sort is an improvement.
>
> I'll try to keep an eye on it over the summer and see if I can't help
> nudge the block layer documents along the path to something unified.
> I've written one custom extension now, so what's a few more?
>
> I also wrote a hasty script that tried to unify the qemu-img.texi and
> qemu-img help outputs and removed one more usage of the hxtool, but it's
> only a stopgap and wasn't something like proper sphinx integration, but
> maybe I could keep working in that area soon.

Thanks.

> I would also like to say that it would be useful to host our HTML docs
> online such that we can point to the different versions, like you can
> with the official python docs. They often have a little dropdown where
> you can switch between 3.5/3.6/3.7 or so on and see the difference on
> that exact page. It'd just simply be nice to be able to do the same for
> QMP docs.

Yes, definitely, although this is I think somewhat orthogonal to
the format we use to create them.

thanks
-- PMM