Re: [PATCH 5/8] parsers: support translatable token aliases

[Adrian Vogelsgesang]

Another related topic:
How should we do the extraction of all translatable strings?

# The problem

Not sure how other translation workflows might look, but in our case we parse 
all our .cpp files to
find all calls to the "translate" function and extract their parameters. We 
could easily adjust this to
also look for calls to the "_" function.

However, our current extractor only works if the function is called with a 
literal string:
> _("my message")
would work, but
> yytranslatable[yytoken] ? _(yytname_[yytoken]) : yytname_[yytoken];
as generated in the lalr1.cc would be an issue for our extractor because " 
yytname_[yytoken]" is
not a string literal but a variable.

We would probably have to parse the grammar file itself in order to extract the 
translatable strings.
Our extractor is currently only capable of parsing C++ and adding parsing 
support for grammar files
seems a bit over the top.

I am not sure how bison does extraction itself. It seems to be using xgettext 
which parses the ".y" file
as if it was a C file and this works fine.

# Proposed solution

The easiest solution would of course be to just use `grep` on the grammar file 
to find
all "_" calls and extract their parameters.

However, this is a bit hacky and, given that bison itself already has a 
complete list
of all tokens, it would be nice to have something like
> bison --extract-translatable-tokens grammar.ypp
to let bison dump a `\n` (or `\0`?) separated list of translatable strings to 
the console.

I could imagine that other users with other translation workflows have similar 
issues
extracting the translatable strings, so it might make sense to integrate this 
functionality
directly into bison.

What are your thoughts on this? 

Cheers,
Adrian


On 02/01/2019, 13:40, "Adrian Vogelsgesang" <address@hidden> wrote:

    Hi Akim,
    
    That's perfect for our use case of bison.
    We have our own translation infrastructure in place and being able to plug 
this into the "_" function will be pretty straightforward.
    At first, I was concerned that "_" might be hard-coded, e.g. to gettext.
    Happy to hear that this is not the case and we can wire up our own function.
    
    Best,
    Adrian
    
    From: Akim Demaille <address@hidden>
    Date: Wednesday, 2 January 2019 at 12:20
    To: Adrian Vogelsgesang <address@hidden>
    Cc: "address@hidden" <address@hidden>, Rici Lake <address@hidden>, Paul 
Eggert <address@hidden>
    Subject: Re: [PATCH 5/8] parsers: support translatable token aliases
    
    > Le 2 janv. 2019 à 09:51, Adrian Vogelsgesang <address@hidden> a écrit :
    > 
    > Hi Akim,
    
    Hi Adrian!
    
    > nice patch series! Thanks for the great work.
    
    You are most welcome :)
    
    > Being able to translate token names will be definitely useful and I am 
looking forward to using that functionality.
    > 
    > I just took a short glance over the patches and maybe I am missing 
something, but:
    > Where is the `_` function used by http://lalr1.cc defined?
    > Am I supposed to put a "#define _(T) myTranslateFunc(T)" into my grammar 
or is it already defined somewhere?
    
    Yes, the user is in charge of providing the translation infrastructure. We 
cannot use the one bison generates in the parsers (YY_), since they use a 
specific catalogue (= set of translations) for yyparse, which contains "parse 
error", etc.
    
    So you have to provide a _ function, which takes a string and returns its 
translation. In typical situations (such as bison's own parser) the remainder 
of the program has translation support, and you just have to extend it to the 
parser. In the case of bison itself, there was nothing to do, as the parser 
already used _ for some custom messages.
    
    I'll put all this in docs once there is some consensus on this branch and 
its alternatives.