Re: Shrinking the C core

[Arthur Miller]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>>>> IMNSHO, discussing a rewrite of Emacs in _any_ language is waste of
>>>> time and energy.  We've seen this many times (because people still
>>>> insist on bringing this up from time to time).  From where I stand,
>>>> the main reason is not even the fact that we decided not to do that,
>>>> but the fact that such a rewrite will never happen in practice.  Such
>>>> a rewrite is a massive job which requires very good knowledge of Emacs
>>>> internals and features, and a lot of time.  People who come close to
>>>> the required knowledge level are not interested in doing this job
>>>> (because they understand the futility), and those who think it should
>>>> be done simply don't know enough and/or don't have enough time on
>>>> their hands to pull it through.
>>>>
>>>> If Emacs will ever be "rewritten", it will not be Emacs, but a
>>>> text-processing system with a very different architecture and design,
>>>> which will take from the Emacs experience the lessons we learned and
>>>> implement them differently, to produce a system whose starting point
>>>> is closer to the needs of today's users and whose main technologies
>>>> are more modern from the get-go.
>>>
>>>I couldn't agree more.
>>>
>>>To me, a rewrite is quatsch, while adding CL facilities to Emacs makes a
>>>lot of sense.
>>
>> I use to say often: either CL will come to Emacs or Emacs to CL, whichever
>> way around. We need some of features available on CL platforms, sbcl
>> notably: built-in concurrency and better garbage collectors from the get-go; 
>> and
>> some of the CL language features, namespaces notably, would be very nice to
>> have.
>
> I agree.  Alas, others, who haven't seem the light yet, don't :-).
>
>> I am not sure which one is easier to achieve, porting elisp to cl, or
>> rewriting core to have all those features.
>
> I don't know either, of course. I guess it depends on the feature.  Some
> random thoughts:
>
> I'm pretty sure that CL packages could be added to Emacs as it is, if
> some people would work on it.

With "CL packages" you mean namespaces? There seem to be already a
branch that implements them, but I don't know how well it works, I
haven't tried it. I think the reasoning is "with don't want them because
we have done 40 years without". But IDK for sure. If there is some
technical reason, then they could help the author to implement it the
way they like it technically.

> I'm also pretty sure that an incremental + generational GC could be
> added, at least as an option, because I would have almost done it some
> 20+ years ago.  It was torpedoed by a patent issue concerning
> mostly-copying GC.  The patent has since expired. A lot of work, of
> course.  I think some people do or have done something in this area, but
> I don't know details.

I am not very familiar wth GC:s implementation more then just some
bired-eye overview. SBCL recently started to move towards non-moving
GC to help with the speed, notably when calling native code which does
not like it's pointers moved underneath, but I am not expert on details
there, these what I have got from the paper:

https://applied-langua.ge/~hayley/swcl-gc.pdf

> I'm not at all sure that non-cooperative multi-threading could be added
> to Emacs.  But I'm also not sure how a CL core would help here.

They are exposing posix threads and have done some work to make at least
parts of the Lisp system work well with threads, and it seems it is
working well for many applicaitons.

http://www.lichteblau.com/sbcl/doc/manual/sbcl/Implementation-_0028Linux-x86_0029.html#Implementation-_0028Linux-x86_0029

I think there is also a missconception in Emacs community that Emacs
loop itself has to be parallelized; I am not sure it is needed; I think
for many people it would be enough to expose threading in form of "js
workers" or something like that. It can be done with processes of
course, but people seem to constantly scream about it in disucssions. CL
has things like lparallel and green threads built on top of hardware
threads, so even there is a bit of job already done. I am sure all that
can be done in Emacs too, but I think, both communities would be more
helped if we perhaps used sbcl and interested individuals helped make
sbcl runtime better intead of reduplication the entire effort.

> On the other hand, I'm pretty convinced that an Emacs core written in CL
> would have to be close to 100% compatible with the existing C core to be
> accepted by users.  That includes a CL rewrite of the C Elisp, including
> byte code interpreter.

Yes, my conclusion too.

> accepted by users.  That includes a CL rewrite of the C Elisp, including
> byte code interpreter.

I am not sure how much of byte-interpretter is needed; I was thinking
how byte interpretter and native compiler fitt there. Oviously since
sbcl is a compiler, with don't need all that stuff, but I am not sure
how much of byte code intepretter is needed. I am sure we need to
understand all of the syntax, since byte code is a valid elisp,
according to the manual; so the reader have to be able to read the
syntax I guess, as printed representation, and has to print same stuff
back to feed into elisp functions.

> That's a massive endeavor.  My hair stands up when I remember the
> compatibility problems I faced with the new redisplay ages ago.
> Multiply that by some factor > 1.  But maybe that's a burnt child
> dreading the fire :-).

Yes, I know. I am fully aware that it is an impossibility for someone
alone, even for a very few. I don't think it is a burnt child, since
yes, the character renderer of Emacs has to be implemented if Emacs
applications will run unchanged.

Hopefully it will be possible to implement Emacs stuff as a special kind
of terminal/character renderer over some sort of tree/graph structure. I
think CLOS and CL have much better tools to refactor that stuff than C,
but what do I know, I haven't tried that and I am not sure if I will
tbh. I am bolling with the ideas. I have seen what they do in other
similar CL software (Hemlock, McClIM, Lem), perhaps there is something
that can be reused there, but I don't know how much and what yet. And,
yes text properties are a special chapter on its own :).

> I'm also not sure how a CL (not Elisp) program would look like using a
> CL Emacs core.

It would look exactly the same, that is the point :). If I wan't to be
able to run elisp applications from elpa/melpa the elisp compatibility
has to be 1:1 so to say. I think it is OK to have some "special places",
like an empty eieio.el, and bunch of aliases or compiler macros for cl-*
stuff, but in general it should be able to read an elisp file without
mofications. 

> CL Emacs core.  Is it nice enough, so to speak?  Think of Emacs strings,
> which couldn't be CL strings because of text properties, buffer-local
> variables...

Yes, buffer locals have to be implemented in CL, thanks to namespaces we
can hav a cl:string and el:string, of which el:string can be a struct
with interval and a list of properties or some other I don't know. The
hairy stuff with string properties is splitting of intervals and
properties and so on. There is much more than just those. I don't know
yet how efficient it would, I plan to test with generic functions. There
are some people testing with static dispatch for generic functions, but
I am not sure if that would work or not yet (works only for compile time
strings), I haven't had time to experiment yet.

> (Another ansatz might be to make Emacs C core a lib.  I haven't given
> that much thought, but it could be more promising than rewriting the
> whole shit in CL :-).)

Perhaps, but I don't think I personally care longer. Once you test and
learn how nice it is to be able to just eval a function and test the
change, I am not going back to hackig C again, at least not in my spare
time. Life is too short for that.

>> CFFI would also be nice to have so
>> that users can extend Emacs themselves with other libraries and not have to 
>> wait
>> for the core devs to do it for them. That would also lessen the burden on
>> maintaining that stuff in the core.
>
> FFI for Emacs once existed, I think Dave Love wrote one, for instance.
> Don't know what became of that.  Might be an issue with interfacing to
> non-free libs.

I didn't know it ever existed in Emacs. 

I though it was a political decision to not have it in Emacs. Which I
understand completely, but after seeing the recent software development,
I believe is hurting "us" much more than "them". As I said in some mail
in the context of keyword args, entire life is a struggle between
choices to maximize the gain. In this case I believe our net loss is
much bigger than gain. Interesting to note is that other GNU projects
are not have political dilemmas about CFFI, for example GCL (Gnu Common
Lisp) or GNU Guile.

I am also aware of emacs-ffi via modules:

https://github.com/tromey/emacs-ffi

But nobody seems to be using it since modules are not big thing in Emacs
community so people barey know about it. That can probably change if
someone wrote a "kille app", or in other words an interesting enough
package that uses it.

Yet another hand, that I haven't seen anywhere, would to load sbcl via
compiled module. Sbcl got ability to save itself as a loadable library,
so it would be possible to basically expose Emacs runtime to sbcl and
export some interface to work with it from within Emacs. I am not sure
if that would be a big win over current way of working via external
process and sockets, but an idea perhaps? I don't have time to try it
myself.