Re: [Qemu-devel] QMP Example Formatting in ReST, Sphinx, and Pygments (was: Re: [Bug] Docs build fails at interop.rst)

[Eduardo Habkost]

On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote:
> 
> 
> On 5/20/19 12:37 PM, John Snow wrote:
> > 
> > 
> > On 5/20/19 7:30 AM, Aarushi Mehta wrote:
> >> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
> >> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
> >> (Rawhide)
> >>
> >> uname - a
> >> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
> >> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
> >> allows for the build to occur
> >>
> >> Regards
> >> Aarushi Mehta
> >>
> >>
> > 
> > Ah, dang. The blocks aren't strictly conforming json, but the version I
> > tested this under didn't seem to care. Your version is much newer. (I
> > was using 1.7 as provided by Fedora 29.)
> > 
> > For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
> > which should at least turn off the "warnings as errors" option, but I
> > don't think that reverting -n will turn off this warning.
> > 
> > I'll try to get ahold of this newer version and see if I can't fix it
> > more appropriately.
> > 
> > --js
> > 
> 
> ...Sigh, okay.
> 
> So, I am still not actually sure what changed from pygments 2.2 and
> sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx
> by default always tries to do add a filter to the pygments lexer that
> raises an error on highlighting failure, instead of the default behavior
> which is to just highlight those errors in the output. There is no
> option to Sphinx that I am aware of to retain this lexing behavior.
> (Effectively, it's strict or nothing.)
> 
> This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the
> build works with our malformed json.
> 
> There are a few options:
> 
> 1. Update conf.py to ignore these warnings (and all future lexing
> errors), and settle for the fact that there will be no QMP highlighting
> wherever we use the directionality indicators ('->', '<-').
> 
> 2. Update bitmaps.rst to remove the directionality indicators.
> 
> 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON.
> 
> 4. Update bitmaps.rst to remove the "json" specification from the code
> block. This will cause sphinx to "guess" the formatting, and the
> pygments guesser will decide it's Python3.
> 
> This will parse well enough, but will mis-highlight 'true' and 'false'
> which are not python keywords. This approach may break in the future if
> the Python3 lexer is upgraded to be stricter (because '->' and '<-' are
> still invalid), and leaves us at the mercy of both the guesser and the
> lexer.
> 
> I'm not actually sure what I dislike the least; I think I dislike #1 the
> most. #4 gets us most of what we want but is perhaps porcelain.
> 
> I suspect if we attempt to move more of our documentation to ReST and
> Sphinx that we will need to answer for ourselves how we intend to
> document QMP code flow examples.

Writing a custom lexer that handles "<-" and "->" was simple (see below).

Now, is it possible to convince Sphinx to register and use a custom lexer?

$ cat > /tmp/lexer.py <<EOF
from pygments.lexer import RegexLexer, DelegatingLexer
from pygments.lexers.data import JsonLexer
import re
from pygments.token import *

class QMPExampleMarkersLexer(RegexLexer):
    tokens = {
        'root': [
            (r' *-> *', Generic.Prompt),
            (r' *<- *', Generic.Output),
        ]
    }

class QMPExampleLexer(DelegatingLexer):
    def __init__(self, **options):
        super(QMPExampleLexer, self).__init__(JsonLexer, 
QMPExampleMarkersLexer, Error, **options)
EOF
$ pygmentize -l /tmp/lexer.py:QMPExampleLexer -x -f html <<EOF
    -> {
         "execute": "drive-backup",
         "arguments": {
           "device": "drive0",
           "bitmap": "bitmap0",
           "target": "drive0.inc0.qcow2",
           "format": "qcow2",
           "sync": "incremental",
           "mode": "existing"
         }
       }

    <- { "return": {} }
EOF
<div class="highlight"><pre><span></span><span class="gp">    -&gt; 
</span><span class="p">{</span>
         <span class="nt">&quot;execute&quot;</span><span class="p">:</span> 
<span class="s2">&quot;drive-backup&quot;</span><span class="p">,</span>
         <span class="nt">&quot;arguments&quot;</span><span class="p">:</span> 
<span class="p">{</span>
           <span class="nt">&quot;device&quot;</span><span class="p">:</span> 
<span class="s2">&quot;drive0&quot;</span><span class="p">,</span>
           <span class="nt">&quot;bitmap&quot;</span><span class="p">:</span> 
<span class="s2">&quot;bitmap0&quot;</span><span class="p">,</span>
           <span class="nt">&quot;target&quot;</span><span class="p">:</span> 
<span class="s2">&quot;drive0.inc0.qcow2&quot;</span><span class="p">,</span>
           <span class="nt">&quot;format&quot;</span><span class="p">:</span> 
<span class="s2">&quot;qcow2&quot;</span><span class="p">,</span>
           <span class="nt">&quot;sync&quot;</span><span class="p">:</span> 
<span class="s2">&quot;incremental&quot;</span><span class="p">,</span>
           <span class="nt">&quot;mode&quot;</span><span class="p">:</span> 
<span class="s2">&quot;existing&quot;</span>
         <span class="p">}</span>
       <span class="p">}</span>

<span class="go">    &lt;- </span><span class="p">{</span> <span 
class="nt">&quot;return&quot;</span><span class="p">:</span> <span 
class="p">{}</span> <span class="p">}</span>
</pre></div>
$ 


-- 
Eduardo