Re: [Qemu-devel] QMP Example Formatting in ReST, Sphinx, and Pygments (was: Re: [Bug] Docs build fails at interop.rst)

[John Snow]

On 5/20/19 7:04 PM, Eduardo Habkost wrote:
> On Mon, May 20, 2019 at 05:25:28PM -0400, John Snow wrote:
>>
>>
>> On 5/20/19 12:37 PM, John Snow wrote:
>>>
>>>
>>> On 5/20/19 7:30 AM, Aarushi Mehta wrote:
>>>> https://paste.fedoraproject.org/paste/kOPx4jhtUli---TmxSLrlw
>>>> running python3-sphinx-2.0.1-1.fc31.noarch on Fedora release 31
>>>> (Rawhide)
>>>>
>>>> uname - a
>>>> Linux iouring 5.1.0-0.rc6.git3.1.fc31.x86_64 #1 SMP Thu Apr 25 14:25:32
>>>> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> Reverting commmit 90edef80a0852cf8a3d2668898ee40e8970e431
>>>> allows for the build to occur
>>>>
>>>> Regards
>>>> Aarushi Mehta
>>>>
>>>>
>>>
>>> Ah, dang. The blocks aren't strictly conforming json, but the version I
>>> tested this under didn't seem to care. Your version is much newer. (I
>>> was using 1.7 as provided by Fedora 29.)
>>>
>>> For now, try reverting 9e5b6cb87db66dfb606604fe6cf40e5ddf1ef0e7 instead,
>>> which should at least turn off the "warnings as errors" option, but I
>>> don't think that reverting -n will turn off this warning.
>>>
>>> I'll try to get ahold of this newer version and see if I can't fix it
>>> more appropriately.
>>>
>>> --js
>>>
>>
>> ...Sigh, okay.
>>
>> So, I am still not actually sure what changed from pygments 2.2 and
>> sphinx 1.7 to pygments 2.4 and sphinx 2.0.1, but it appears as if Sphinx
>> by default always tries to do add a filter to the pygments lexer that
>> raises an error on highlighting failure, instead of the default behavior
>> which is to just highlight those errors in the output. There is no
>> option to Sphinx that I am aware of to retain this lexing behavior.
>> (Effectively, it's strict or nothing.)
>>
>> This approach, apparently, is broken in Sphinx 1.7/Pygments 2.2, so the
>> build works with our malformed json.
>>
>> There are a few options:
>>
>> 1. Update conf.py to ignore these warnings (and all future lexing
>> errors), and settle for the fact that there will be no QMP highlighting
>> wherever we use the directionality indicators ('->', '<-').
>>
>> 2. Update bitmaps.rst to remove the directionality indicators.
>>
>> 3. Update bitmaps.rst to format the QMP blocks as raw text instead of JSON.
>>
>> 4. Update bitmaps.rst to remove the "json" specification from the code
>> block. This will cause sphinx to "guess" the formatting, and the
>> pygments guesser will decide it's Python3.
>>
>> This will parse well enough, but will mis-highlight 'true' and 'false'
>> which are not python keywords. This approach may break in the future if
>> the Python3 lexer is upgraded to be stricter (because '->' and '<-' are
>> still invalid), and leaves us at the mercy of both the guesser and the
>> lexer.
>>
>> I'm not actually sure what I dislike the least; I think I dislike #1 the
>> most. #4 gets us most of what we want but is perhaps porcelain.
>>
>> I suspect if we attempt to move more of our documentation to ReST and
>> Sphinx that we will need to answer for ourselves how we intend to
>> document QMP code flow examples.
> 
> Writing a custom lexer that handles "<-" and "->" was simple (see below).
> 
> Now, is it possible to convince Sphinx to register and use a custom lexer?
> 

Spoilers, yes, and I've sent a patch to list. Thanks for your help!