Use "simple parsers" for translations again?

[Gavin Smith]

On Sun, Nov 05, 2023 at 12:53:19PM +0100, Patrice Dumas wrote:
> Indeed, I don't know why I want it to have an effect, as it should
> not...
> 
> > If there is a real problem, then we could fix it another way, maybe
> > by "locking" the parser configuration at certain points, making any
> > changes ineffective?
> 
> There is a real problem, the state of the parsers is shared among calls
> to simple_parser.  As long as in gdt code there is only simple code it
> is not an issue, though, it only becomes an issue if the users put
> @-commands that use the conversion state in gdt.  In that regard, the
> translation_in_parser_in_translation test is quite extreme, but
> @-commands modifying and needing the state are not so uncommon, it
> includes commands that could be in translations, such as @acronym,
> @anchor and others.

As I understand it, it is not any part of the state that is problematic
to modify for simple_parser.  It is only the structures that are referred
to when creating the simple parser:

  $parser->{'line_commands'} = $simple_parser_line_commands;
  $parser->{'brace_commands'} = $simple_parser_brace_commands;
  $parser->{'valid_nestings'} = $simple_parser_valid_nestings;
  $parser->{'no_paragraph_commands'} = $simple_parser_no_paragraph_commands;
  $parser->{'index_names'} = $simple_parser_index_names;
  $parser->{'command_index'} = $simple_parser_command_index;
  $parser->{'close_paragraph_commands'} = 
$simple_parser_close_paragraph_commands;
  $parser->{'close_preformatted_commands'} = 
$simple_parser_close_preformatted_commands;

Many of these relate to the creation of new Texinfo commands, or indices.
These structures should be the same for all of the parser objects used
for translations so copying them (with dclone) for every single translated
string is a waste of time.

It seems that we could stop these structures being modified with the
addition of a flag on the parser (e.g. $parser->{'locked'} or
$parser->{'simple'}).  Then check this flag any time:

* A new Texinfo command is to be defined (@macro, @alias, @definfoenclose).
* A new index is to be added (@defindex, @defcodeindex).

Then the change to the state wouldn't be made if the flag was set, and
possibly a warning message could be given too.

I can try to work on a patch for this if this approach sounds like a
good idea.

> I think that the approach of having shared state is both unusual and not
> correct. Another possibility, which could be better and allow some gains
> too would be to reuse a parser instead without initializing it.  It
> would still maybe need some more cleanup of state between runs, maybe
> with a separate function that would do nothing in XS, such that it does
> not need to be run when the parser is not reused.  This would also
> probably be more similar to what should be the final XS implementation.

Reusing the same parser object, with cleanup in between, seems to me to
be even less correct as it means that the whole parser object is being
shared across translations, rather than just certain parts of it, making
it more likely that something persists between translations (e.g. @anchor
locations).

> 
> > I would like to commit this, except that the
> > "perl -w t/init_files_tests.t translation_in_parser_in_translation"
> > test breaks.
> 
> It will break other redefinitions of translated strings.

I've still not had the time to investigate what is going on with this
test.  Hopefully I'll get to it this weekend.

In general, it seems like we're making the program significantly slower
for a theoretic and very unlikely use case.