Re: Question about display engine

[Eli Zaretskii]

> Cc: address@hidden, address@hidden
> From: martin rudalics <address@hidden>
> Date: Tue, 13 Aug 2019 10:17:32 +0200
> 
> > (And shouldn't we simply disregard entirely a face whose :extend
> > attribute is nil?  What can such a face possibly contribute in this
> > case?)
> 
> Consider a user who sets the :extend attribute to non-nil for the
> 'default' face and wants a 'link' background to not extend to the end
> of line.  Such a user would want to set the :extend attribute of the
> 'link' face to nil to get the desired effect.

Yes, and my point is that in that case we can simply ignore the 'link'
face when extending the face of the last character of a line.

> > You seem to assume that the iterator examines faces at every buffer
> > position it passes, and in particular on the newline at EOL.
>
> I certainly do not assume such a thing.  But when trying to find out
> how face attributes are merged and realized, face_at_buffer_position
> was the first point of reference where I found that (and I still don't
> know a better one).

face_at_buffer_position is the right place, but it isn't called for
every buffer position.  And your description seemed to hint that you
thought it was called for every buffer position.

> > But
> > that's not what happens, because examining and merging faces is
> > expensive, so Emacs avoids doing that unless necessary.  We only
> > examine faces where they change, and use next-single-property-change
> > to find the next position where we should again examine the faces.
> 
> That's exactly what I expected but did not find immediately.  If you
> tell me where to look for that (Fnext_single_property_change _is_
> called by face_at_buffer_position) "examine faces where they change"

This has to do with how the display iterator works in general.  It
starts by determining where the next "stop position" is.  A "stop
position" (stored in it->stop_charpos) is the position where iteration
must stop and examine all the possible sources of affecting what and
how should be displayed next.  This includes faces, on-the-fly
fontification, invisible text, display properties, overlays, switching
from iteration over a buffer to iteration of a string and back, etc.
At that stop position, the iterator examines all these potential
game-changers, and computes and stores the results in its fields.
That is where face_at_buffer_position is called, and the result is
it->face_id, a numerical index into the frame's face cache which
defines the realized face to be used from now on for displaying
characters.  The iterator will not call face_at_buffer_position and
won't consider faces until it gets to the next stop position.

The next stop position is computed in compute_stop_pos.  You will see
that it doesn't call Fnext_single_property_change, but instead
accesses the interval tree directly, because that's more efficient:
you find changes in _any_ text properties that way.  It also considers
overlay changes and changes in character composition.  The next stop
position is the closest one of those provided by any of these sources.

When we get to the stop position, we invoke a series of known
handlers, each one responsible to handle the relevant Lisp data
structures that might affect the display.  See handle_stop.  In
particular, the handler of face properties is handle_face_prop, which
calls face_at_buffer_position or face_at_string_position.  Each
handler updates the relevant fields of the iterator structure as
necessary.  Having handled everything that needs to be handled at a
particular stop position, we call compute_stop_pos to compute the next
stop position, and then proceed producing glyphs using the updated
iterator information until we get to that next stop position, never
examining any of the above factors again until we get to that place.

> and in particular where a :background change is found and applied, I
> probably could tell you more.

Neither :background nor any other face attribute is "found and
applied" in the main iterator loop.  The iterator simply records in
each glyph the face ID to be used for displaying that glyph.  That
face ID is the value returned by the last call to
face_at_buffer_position.  The code which examines these attributes is
in the terminal-specific backends: xterm.c, w32term.c, nsterm.m, etc.
Only there we extract the colors, the underline, the strike-through,
etc. attributes, and use them to produce the corresponding visual
appearance.  For example, for the background color, that is where we
instruct the GUI system to "clear to EOL" with that color.

(The last paragraph is slightly over-simplified: you will see in a
couple of places that we do look at face->background in xdisp.c,
notably in extend_face_to_end_of_line, but only in order to compare it
with the frame's default background.)

> > In addition, face_at_buffer_position doesn't look for face attributes
> > one by one.  Instead, it finds all the source of the 'face' property
> > that are in effect at the given position -- the default face, the face
> > from text properties, and from all the overlays at that position --
> > and merges these attributes into a single attribute vector.  Then it
> > looks up a realized face with identical attributes, or realizes a new
> > face if no such face exists.  Thereafter, the iterator just uses that
> > face until the next checkpoint.  IOW, face_at_buffer_position returns
> > an ID of a realized (i.e. fully specified) face created by merging
> > several relevant sources of face information, and that realized face
> > has no references to the names of the individual faces from which it
> > was created, nor any memory of which non-unspecified attributes came
> > from which face source.
> 
> When face_at_buffer_position puts some 'background' value into that
> single attribute vector, it can simply set an 'extend_background' bit
> in that vector which tells the display engine whether the background
> specified by that vector shall be extended or not.

So you propose to have an "extend" bit for every face attribute?
There are 18 of them.  (Maybe some of the attributes won't need that
bit, but quite a few will.)

Moreover, given the description above of where the attributes are
acted upon, you are saying that xterm.c, w32term.c etc. will have to
consider whether the glyphs they display are at the end of a line or
not, and act accordingly, i.e. ignore some of the face attributes
under certain conditions, right?

> >   . modify face_at_buffer_position and its subroutines to behave
> >     specially when called on a newline,
> 
> I strongly doubt that this "called on a newline" is needed.  Setting
> the 'extend_background' indicator is strongly tied to the last setting
> of the 'background' indicator in the attribute vector as done by the
> face merging algorithm.

But the merged face will be used for displaying both the "extension"
part of the line and "normal" characters.  When displaying "normal"
characters, the background color should be used unconditionally,
whereas while displaying the "extension" part it should be used only
if its 'extend' bit is set.  Right?  What I wrote above assumed that
we make these decisions at face merge time, and will actually have 2
different realized faces; what you are suggesting is that we use a
single realized face and make those decisions when we use the faces to
write to the glass.  That doesn't strike me as a simpler
implementation.

> > Sounds like fun project.  Volunteers are welcome.
> 
> If all these were really necessary, I'd never vote for such a change.

I'm afraid I still don't see a simple solution.  Hopefully someone
will show me the light.