Re: [Texmacs-dev] Review Request: Use blanks, escaped char, inline comment and number to simplify code

[TeXmacs]

Hi Darcy,

On Mon, Jan 06, 2020 at 12:44:05AM +0800, Darcy Shen via Texmacs-dev wrote:
> >Thanks for your parsing patch. It seems like a good idea indeed to
> >have a more systematic way to parse programming languages.
> >I quickly went through your code and noticed a few things:
> >
> >- You often use the list<...> template when you really should be
> >using the array<...> template (which is more efficient for arrays ;^)
> >
> >- You may simplify some lines such as
> >
> >+ if (blanks_parser.parse (s, pos)) {
> >+ break;
> >+ }
> >
> >into
> >
> >+ if (blanks_parser.parse (s, pos)) break;
> >
> >As a general rule, I am happy to see that you adhered quite well to
> >my coding style. This makes it much easier for me to read your code ;^)
> 
> I will improve the code later.

It would be great not to wait too long, now that you did the commit...

> >Do you want me to apply the patch before version 2.1, or do you prefer
> >me to wait a bit? I just got good news from my employer that I will
> >be allowed to auto-edit my book on TeXmacs. I still have to go through
> >some legal business, but TeXmacs 2.1 is definitely approaching fast now.
> 
> Sorry, I have committed the patch.  Well, I thought I waited a week
> but did not
> 
> notice that 2020/01/01 is Wednesday.
> 
> Good news for your book on TeXmacs!
> 
> For TeXmacs 2.1, is TeXmacs 2.1 right after TeXmacs 1.99.12?

There may still be one or two intermediate releases.
I would appreciate if you could refrain from starting
new modifications or code additions.  I now want to focus on
cleaning things up and correcting bugs that might still come up.
I will post an official message sometime soon.

> >One other thing that we have to address at a certain point is
> >coloring schemes.
> >There is quite a lot of ad hoc coloring going on in TeXmacs
> >(syntax highlighting,
> >but also the color of the cursor, selections, the surroundings of
> >the window, etc.).
> >It would be nice to make all this stylable and make it part of the
> >infra-structure
> >for themes that I have been starting to design.
> 
> Good idea!
> 
> Currently, I am still busy on improving the code base of the current
> xxx_language.cpp.
> 
> Find the common routines and do some abstractions. I submitted the
> issue for a long
> 
> time: https://savannah.gnu.org/bugs/?54888
> 
> Finally, there will be no xxx_language.cpp, we wrote the base
> parsers in cpp and combines
> 
> the parsing components in scheme. To support a new language, we only
> need to write a new scheme file.

I am not sure that I entirely follow you, but if you wish to
make things scriptable via Scheme, then another option is to
use the integrated packrat parser.  It is not as efficient as
native C++ (although still reasonably efficient) and
it already supports syntax highlighting.  It should not be very
difficult to add what is needed for hyphenation and spacing.
It has the additional benefit that one might actually parse
the programs and produce a syntax tree for them.
It also allows you to make semantic selections.
Finally, you may define sublanguages, a bit like what you
started to do by factoring out common parsing patterns.
You may look at progs/language/minimal.scm for an example and
at std-math.scm for a more sophisticated example that
is used for semantic editing.

The main reason why I refrained from using the packrat tool
so far for programming languages is efficiency and
also because I simply already had something that worked.
Furthermore, packrat parsing is typically more rigid,
since your program is _really_ getting parsed;
of course, nothing withholds you from using a losened grammar.

So if you have some time, then it might be worth it
to experiment with a packrat-based approach, because it
definitely is the most elegant one from a high level point of view.
Maybe a few careful hacks would allow to address efficiency issues.
Note also that if we have all our grammars in packrat form,
then we may always write a another packrat parser or
parsing routine that might be more optimized for programming languages.
The speed issue is not really a conceptual problem.

Best wishes, --Joris