Re: Ligatures (was: Unify the Platforms: Cairo+FreeType+Harfbuzz Everywhere (except TTY))

[Pip Cet]

On Sat, May 23, 2020 at 5:32 PM Eli Zaretskii <address@hidden> wrote:
> > From: Pip Cet <address@hidden>
> > Date: Sat, 23 May 2020 15:13:38 +0000
> > Cc: address@hidden, address@hidden, address@hidden
> >
> > As I said, the problem I have is to get look-ahead working, which you
> > think isn't a problem. I've got an idea for it, but it doesn't work
> > (yet); my theory is the bidi.c code fails to keep its state in the
> > iterator and can't deal with multiple parallel iterators.
> >
> > > I could be wrong, though, so I'm looking
> > > forward to see you present a series of patches that do support the
> > > existing use cases and the ligatures as well, and don't cause any
> > > slowdown in redisplay.
> >
> > As I said, what's stopping me is the look-ahead problem, and in
> > particular some code in bidi.c that doesn't play along well with
> > look-ahead.
>
> I don't think you understand the depth of the issue.

I think I do, actually. It's just that you'd prefer the display engine
to be torn out by the roots and rewritten in one fell swoop, but that
option isn't currently on the table.

> If we are going
> to send large chunks of text to the shaping engine, then none of the
> insane complexity of bidi.c makes sense; we should simply throw all of
> it away and use a very different, batch-style reordering code, of the
> kind you can find in the freebidi library.  The sole reason for
> bidi.c's existence is to produce character positions in the _visual_
> order, one position at a time, something that no other bidi-aware
> editor does.

Yes, except we're not talking about "large chunks of text" here:
somehow you went from "we need only a bunch of regexps to catch
ligatures" to "we need to send entire paragraphs to the shaping
engine, nothing less will do". My opinion is that we need a reasonable
amount of context, often just a single character, and I see no reason
to throw out the entire display engine because we want some
look-ahead.

> Moreover, not even the basic iteration, one level above bidi.c, where
> we call get_next_display_element, then PRODUCE_GLYPHS, then
> set_iterator_to_next -- not even that makes sense.

Again, a single character of lookahead in the typical case, four
characters for most ligatures; that doesn't push us over the line to
"only a complete rewrite makes sense".

> This basic loop
> exists only because we examine characters one by one, switching from
> buffer text to overlay or display strings, then back, as needed, and
> applying faces as we go.  Doing this in large chunks calls for a very
> different structure of the code, and very different separation into
> layers.

Indeed. Which is why I'm not talking about doing it in large chunks,
at this point. Let's keep doing it character by character but add what
little we need to in order to look ahead a little.

> This needs to be carefully designed in advance in a clean and
> well-defined way, not lumped one patch upon another until it kinda
> works...

I agree "just start hacking on it with no understanding of the code
until things appear to start working" is a bad strategy. So is "first,
redesign the universe". To me, it seems like what I want is a
reasonable compromise: not large chunks of text, because we can't do
that, but some context, enough to do kerning and deal with ligatures.

Remember that this discussion started when I mentioned that I was
unhappy with HarfBuzz, and I still am, precisely because of its
"first, send me your entire document" approach. I don't think it's the
right approach to take this design flaw of HarfBuzz for granted and
conclude that we need to rewrite the Emacs display engine before we
can get English ligatures to display properly.

If, that is, we can get look-ahead to work.