Re: Thoughts on getting correct line numbers in the byte compiler's warning messages

[Alan Mackenzie]

Hello, Stefan.

On Tue, Nov 06, 2018 at 08:56:48 -0500, Stefan Monnier wrote:
> > Actually this idea was not good;

> [ I'll assume you're not talking about the idea of using such a reader in
>   edebug, but about using such a reader for your use case.  ]

In particular, in the byte compiler.

> > macros could not handle such a form without severe changes in the way
> > macros work.  (A research project, perhaps).

> Right.  The way I was thinking about it was that when calling
> macros we'd do something like:

>     (plain-to-annotated
>      (macroexpand (annotated-to-plain sexp)))

That would lose too much of the wanted source position data.

> not a research project by any stretch, but its impact on performance
> could be a problem, indeed.

> > The reader would produce, in place of the Lisp_Objects it currently
> > does, an object with Lisp_Type 1 (which is currently unused).  The rest
> > of the object would be an address pointing at two Lisp_Objects, one
> > being the "real" read object, the other being a source position.

> More generally, you're suggesting here to add a new object type (could
> just as well be a new pseudo-vector or any such thing: these are just
> low-level concerns that don't really affect the overall design).

There's nothing just about hurting performance.

> > The low level routines, like CONSP, and a million others in lisp.h would
> > need amendment.

> So you're suggesting to change the low-level routines accessing
> virtually all object types to also accept those "annotated objects"?

Yes.

> That means all processing of all objects would be slowed down.
> I think that's a serious problem (I'd rather pay a significant slow
> down in byte-compilation than a smaller slowdown on everything else).

The slow down would not be great.  For example, XCONS first checks the
3-bit tag, and if all's OK, removes it, otherwise it handles the error.  I'm
proposing enhancing the "otherwise" to check for a tag of 1 together
with a proper cons at the far end of a pointer.  With care, there should
be no loss in the usual case, here.

I timed a bootstrap, unoptimised GCC, with an extra tag check and
storage to a global variable inserted into XFIXNUM.  (Currently there is
no such check there).  The slowdown was around 1.3%

> > But the Lisp system would continue with 8-byte objects,
> > and the higher level bits (nearly all of it) would not need changes.
> > The beauty of this scheme is that, outside of byte compilation, nothing
> > else would change.

> Also, I wonder how this (or any other of the techniques discussed) solve
> the original problem you describe:

>     The forms created by the reader go through several (?many)
>     transformative phases where they get replaced by successor forms.
>     This makes things more difficult.

Many of the original forms produced by the reader survive these
transformations.  For those that do not, we could bind
byte-compile-containing-position (or whatever) to a sensible position
each time the compiler enters a "major" form (whatever that might mean).

> E.g. we could implement big-object as

> 1.  (defun big-object (object location)
>       (cons object location))
> or
> 2.  (defun big-object (object location)
>       (puthash object location location-hash-table)
>       object)
> or
> 3.  (defun big-object (object location)
>       (make-new-special-object object location))

1. wouldn't work, as such.  E.g. evaluating `car' must get the car of
the original OBJECT, not the car of (cons OBJECT LOCATION).

I've tried 2., and given up on it: everywhere in the compiler where FORM
is transformed to NEWFORM, a copy of a hash has to be created for
NEWFORM.  Also, there's no convenient key for recording the hash of an
occurence of a symbol (such as `if').

3. is what I'm proposing, I think.  The motivating thing here is that
the rest of the system can handle NEW-SPECIAL-OBJECT and get the same
result it would have from OBJECT.  Hence the use of Lisp_Type 1, or
possibly a new pseudovector type.

> but the problem remains of how to put it at all the places where we
> need it.

Every object produced by the reader during byte compilation would have
its source position attached to the object, in essence.  Objects
produced by macro expansion would not have this, but we could arrange to
copy the info much of the time.  (E.g. the result of a `mapcar'
operating on a list of FORMs would be given the position information of
the list.)  Other non-reader forms would have to depend on the variable
byte-compile-containing-position mentioned above.

Incidentally, I'm coming round the the idea of calling the new object an
_extended_ object.  In place of the fixnum source position proposed, we
could use, for example, a property list.  There are surely many
applications for having a property list on a cons form.  :-)

> > The extra indirection involved in these "big objects" would naturally
> > slow down byte compilation somewhat.  I've no idea how much, but it
> > might not be much at all.

> Indeed, I don't think that's a significant issue.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).