Re: On elisp running native

[Gong-Yi Liao 廖宮毅]

That's pretty good on (I assume) x86_64, but native-comp branch seems
not working on ARM64 Debian Buster at this moment:
Not sure if it's a GCCJIT issue or the native-comp branch issue:

Any comments on this compilation log are appreciated.

Thanks,
Gong-Yi.

------
make -C ../lisp compile-first EMACS="../src/bootstrap-emacs"
make[2]: Entering directory '/home/pi/Downloads/emacs/master/lisp'
 ELC+ELN   emacs-lisp/comp.elc
 ELC+ELN   emacs-lisp/bytecomp.elc
 ELC+ELN   emacs-lisp/autoload.elc
Backtrace:
../src/bootstrap-emacs(+0x1289f0)[0x555c89f9f0]
../src/bootstrap-emacs(+0x46b5c)[0x555c7bdb5c]
../src/bootstrap-emacs(+0x46d50)[0x555c7bdd50]
../src/bootstrap-emacs(+0x12720c)[0x555c89e20c]
../src/bootstrap-emacs(+0x127298)[0x555c89e298]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fa303b698]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc014dc)[0x7f9c8b94dc]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc015b0)[0x7f9c8b95b0]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17c890)[0x7f9be34890]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17ec94)[0x7f9be36c94]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17f5c8)[0x7f9be375c8]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17e760)[0x7f9be36760]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x1749e8)[0x7f9be2c9e8]
/lib/aarch64-linux-gnu/libgccjit.so.0(gcc_jit_context_compile_to_file+0x94)[0x7f9be214ec]
../src/bootstrap-emacs(+0x1b9bbc)[0x555c930bbc]
../src/bootstrap-emacs(+0x18119c)[0x555c8f819c]
../src/bootstrap-emacs(+0x1818ac)[0x555c8f88ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x1825ac)[0x555c8f95ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x18160c)[0x555c8f860c]
../src/bootstrap-emacs(+0x180cc8)[0x555c8f7cc8]
../src/bootstrap-emacs(+0x180f7c)[0x555c8f7f7c]
../src/bootstrap-emacs(+0x181a40)[0x555c8f8a40]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x182864)[0x555c8f9864]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x1825ac)[0x555c8f95ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x18160c)[0x555c8f860c]
../src/bootstrap-emacs(+0x17f0fc)[0x555c8f60fc]
../src/bootstrap-emacs(+0x1812a0)[0x555c8f82a0]
../src/bootstrap-emacs(+0x181a40)[0x555c8f8a40]
../src/bootstrap-emacs(+0x1810c4)[0x555c8f80c4]
../src/bootstrap-emacs(+0x18160c)[0x555c8f860c]
../src/bootstrap-emacs(+0x17f0fc)[0x555c8f60fc]
../src/bootstrap-emacs(+0x17f250)[0x555c8f6250]
../src/bootstrap-emacs(+0x188048)[0x555c8ff048]
../src/bootstrap-emacs(+0x189d10)[0x555c900d10]
../src/bootstrap-emacs(+0x181330)[0x555c8f8330]
../src/bootstrap-emacs(+0x182a90)[0x555c8f9a90]
...
/bin/bash: line 1:   626 Segmentation fault      EMACSLOADPATH=
'../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval
'(setq load-prefer-newer t)' -l comp -f
batch-byte-native-compile-for-bootstrap emacs-lisp/autoload.el
make[2]: *** [Makefile:312: emacs-lisp/autoload.elc] Error 139
make[2]: *** Waiting for unfinished jobs....

Fatal error 11: Segmentation fault
Backtrace:
../src/bootstrap-emacs(+0x1289f0)[0x555c6709f0]
../src/bootstrap-emacs(+0x46b5c)[0x555c58eb5c]
../src/bootstrap-emacs(+0x46d50)[0x555c58ed50]
../src/bootstrap-emacs(+0x12720c)[0x555c66f20c]
../src/bootstrap-emacs(+0x127298)[0x555c66f298]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f8b597698]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc014dc)[0x7f84e154dc]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc015b0)[0x7f84e155b0]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17c890)[0x7f84390890]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17ec94)[0x7f84392c94]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17f5c8)[0x7f843935c8]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17e760)[0x7f84392760]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x1749e8)[0x7f843889e8]
/lib/aarch64-linux-gnu/libgccjit.so.0(gcc_jit_context_compile_to_file+0x94)[0x7f8437d4ec]
../src/bootstrap-emacs(+0x1b9bbc)[0x555c701bbc]
../src/bootstrap-emacs(+0x18119c)[0x555c6c919c]
../src/bootstrap-emacs(+0x1818ac)[0x555c6c98ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x1825ac)[0x555c6ca5ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x18160c)[0x555c6c960c]
../src/bootstrap-emacs(+0x180cc8)[0x555c6c8cc8]
../src/bootstrap-emacs(+0x180f7c)[0x555c6c8f7c]
../src/bootstrap-emacs(+0x181a40)[0x555c6c9a40]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x182864)[0x555c6ca864]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x1825ac)[0x555c6ca5ac]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x18160c)[0x555c6c960c]
../src/bootstrap-emacs(+0x17f0fc)[0x555c6c70fc]
../src/bootstrap-emacs(+0x1812a0)[0x555c6c92a0]
../src/bootstrap-emacs(+0x181a40)[0x555c6c9a40]
../src/bootstrap-emacs(+0x1810c4)[0x555c6c90c4]
../src/bootstrap-emacs(+0x18160c)[0x555c6c960c]
../src/bootstrap-emacs(+0x17f0fc)[0x555c6c70fc]
../src/bootstrap-emacs(+0x17f250)[0x555c6c7250]
../src/bootstrap-emacs(+0x188048)[0x555c6d0048]
../src/bootstrap-emacs(+0x189d10)[0x555c6d1d10]
../src/bootstrap-emacs(+0x181330)[0x555c6c9330]
../src/bootstrap-emacs(+0x182a90)[0x555c6caa90]
...
/bin/bash: line 1:   625 Segmentation fault      EMACSLOADPATH=
'../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval
'(setq load-prefer-newer t)' -l comp -f
batch-byte-native-compile-for-bootstrap emacs-lisp/bytecomp.el
make[2]: *** [Makefile:312: emacs-lisp/bytecomp.elc] Error 139
Backtrace:
../src/bootstrap-emacs(+0x1289f0)[0x55647679f0]
../src/bootstrap-emacs(+0x46b5c)[0x5564685b5c]
../src/bootstrap-emacs(+0x46d50)[0x5564685d50]
../src/bootstrap-emacs(+0x12720c)[0x556476620c]
../src/bootstrap-emacs(+0x127298)[0x5564766298]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fb85f5698]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc014dc)[0x7fb1e734dc]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0xc015b0)[0x7fb1e735b0]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17c890)[0x7fb13ee890]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17ec94)[0x7fb13f0c94]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17f5c8)[0x7fb13f15c8]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x17e760)[0x7fb13f0760]
/lib/aarch64-linux-gnu/libgccjit.so.0(+0x1749e8)[0x7fb13e69e8]
/lib/aarch64-linux-gnu/libgccjit.so.0(gcc_jit_context_compile_to_file+0x94)[0x7fb13db4ec]
../src/bootstrap-emacs(+0x1b9bbc)[0x55647f8bbc]
../src/bootstrap-emacs(+0x18119c)[0x55647c019c]
../src/bootstrap-emacs(+0x1818ac)[0x55647c08ac]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x1825ac)[0x55647c15ac]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x18160c)[0x55647c060c]
../src/bootstrap-emacs(+0x180cc8)[0x55647bfcc8]
../src/bootstrap-emacs(+0x180f7c)[0x55647bff7c]
../src/bootstrap-emacs(+0x181a40)[0x55647c0a40]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x182864)[0x55647c1864]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x1825ac)[0x55647c15ac]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x18160c)[0x55647c060c]
../src/bootstrap-emacs(+0x17f0fc)[0x55647be0fc]
../src/bootstrap-emacs(+0x1812a0)[0x55647c02a0]
../src/bootstrap-emacs(+0x181a40)[0x55647c0a40]
../src/bootstrap-emacs(+0x1810c4)[0x55647c00c4]
../src/bootstrap-emacs(+0x18160c)[0x55647c060c]
../src/bootstrap-emacs(+0x17f0fc)[0x55647be0fc]
../src/bootstrap-emacs(+0x17f250)[0x55647be250]
../src/bootstrap-emacs(+0x188048)[0x55647c7048]
../src/bootstrap-emacs(+0x189d10)[0x55647c8d10]
../src/bootstrap-emacs(+0x181330)[0x55647c0330]
../src/bootstrap-emacs(+0x182a90)[0x55647c1a90]
...
/bin/bash: line 1:   624 Segmentation fault      EMACSLOADPATH=
'../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval
'(setq load-prefer-newer t)' -l comp -f
batch-byte-native-compile-for-bootstrap emacs-lisp/comp.el
make[2]: *** [Makefile:312: emacs-lisp/comp.elc] Error 139
make[2]: Leaving directory '/home/pi/Downloads/emacs/master/lisp'
make[1]: *** [Makefile:827: bootstrap-emacs.pdmp] Error 2
make[1]: Leaving directory '/home/pi/Downloads/emacs/master/src'
make: *** [Makefile:424: src] Error 2

----

On Thu, Mar 5, 2020 at 4:55 AM Adam Porter <address@hidden> wrote:
>
> Hi Andrea,
>
> Andrea Corallo <address@hidden> writes:
>
> > - Finally I've set up a docker image for people willing to test it
> >   without having to configure and compile Emacs and libgccjit.  Now that
> >   I've learned how it works I plan to use it to setup a small CI to keep
> >   an eye that the latest GCC does not break with our build.
>
> Thank you for setting up the Docker image.  I had tried to build the
> native-comp branch on my system but finally was unable to overcome a
> weird error related to libgccjit (probably because I need to upgrade my
> system).  I even tried using Guix to build it, but libgccjit isn't
> packaged in it (and I'm completely new to Guix packaging, so I wasn't
> able to figure out how to adjust the GCC build for that in a reasonable
> amount of time).
>
> I managed to get the Docker image working, and I did a little testing
> with one of my packages, org-ql, which is for searching Org files.  I
> installed it and its dependencies, then used native-comp-async to
> compile all of ~/.emacs.d/elpa.  Then I restarted Emacs to ensure that
> the native libraries were loaded, and finally I ran the following code
> and observed a more than 2x improvement!
>
> #+BEGIN_SRC elisp
>   ;; Reset the cache to force search to run again.
>   (setf org-ql-cache (make-hash-table :test #'equal))
>
>   ;; Set my personal groups (omitted) and enable org-super-agenda-mode
>   ;; to display in groups.
>   (setf org-super-agenda-groups my-groups)
>   (org-super-agenda-mode)
>
>   ;; Load all my Org files and search them.
>   (let* ((enable-local-variables nil)
>          (files (directory-files org-directory 'full (rx ".org" eos))))
>     (dolist (file files)
>       (find-file-noselect file 'nowarn))
>     (benchmark-run-compiled
>         (org-ql-search files
>           ;; The (tags) selector tends to be slow compared to other
>           ;; predicates because of searching the hierarchy repeatedly
>           ;; and tag inheritance, so it makes for an interesting test.
>           '(and (todo) (tags "Emacs"))
>           :super-groups org-super-agenda-groups)))
> #+END_SRC
>
> #+RESULTS:
> | 5.819892666 | 1 | 0.6101534569999956 |  In Emacs 26.3
> | 2.317804428 | 3 | 0.6223487619999997 |  In Emacs 28 with native-comp
>
> Then I tested this more complex query and observed an even more
> impressive speedup of about 3.4x!
>
>   '(and (not (done))
>         (or (habit)
>             (deadline auto)
>             (scheduled :to today)
>             (ts-active :on today)))
>
> #+RESULTS:
> | 8.377843199 | 1 | 0.6277314330000081 |  In Emacs 26.3
> | 2.436048375 | 2 | 0.4278436059999997 |  In Emacs 28 with native-comp
>
> This is very exciting!  The difference between 8.4 seconds and 2.4
> seconds is a huge usability improvement.
>
> And everything seems to work fine!  No bugs, errors, or crashes of any
> kind.
>
> I did encounter a minor issue related to macro-expansion and loading
> that may be my fault.  When I tried to (require 'org-ql) or call a
> function that is autoloaded from it, I got an error saying that the .eln
> file did not provide the org-ql feature.  I surmised that meant that the
> file hadn't been compiled properly, so I looked at the async compilation
> log and saw an error saying that the variable org-ql-predicates was not
> defined.  At this URL you can see a macro definition,
> org-ql--def-plain-query-fn, and the call to it that's wrapped in
> cl-eval-when to ensure it works properly with the byte-compiler:
>
> https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L1671
>
> The macro uses the value of org-ql-predicates, which is defined here:
>
> https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L126
>
> But when the org-ql--defpred macro:
>
> https://github.com/alphapapa/org-ql/blob/06481960764a55a36d886e1db79a58258d5d2ffb/org-ql.el#L140
>
> ...is called, it adds to that variable.  So all of that works with the
> byte-compiler, but apparently not with the native compilation.
>
> To work around it, I wrapped (defvar org-ql-predicates...) in
> eval-and-compile, and then wrapped all of the calls to org-ql--defpred
> in eval-and-compile as well.  Then I deleted the org-ql.eln file and
> recompiled it, restarted Emacs, and then everything worked fine.
>
> I'm guessing that this issue is "native" to the native-compilation and
> might need to be documented in some way, but I wouldn't expect it to
> affect many packages, only ones that do tricky things with variables and
> macro-expansion like this.
>
> So, a few ideas for fixes or improvements:
>
> 1.  I guess the .eln file should not have been produced or left behind
>     in the directory, since there was a compilation error.
> 2.  The compilation error should probably be shown as a warning or
>     something, because it's easily lost in the async compilation log
>     buffer.
> 3.  It would probably be good if the loader fell back to .elc and .el
>     files if the .eln file doesn't seem to work, and showed a warning
>     when doing so.  The library would remain usable in its byte-compiled
>     form, and in the case of a package like this, the user could then
>     report the native-compilation error to the developer.
>
> Other than that minor issue, everything works and feels very fast!  This
> is great!
>
> >   ~eln~ files are compiled in specific sub-folders taking in account
> >   host architecture and current Emacs configuration to disambiguate
> >   possible incompatibilities.
>
> >  As example a file ~.../foo.el~ will compile into something like
> >  ~.../x86_64-pc-linux-gnu-12383a81d4f33a87/foo.eln~.
>
> FYI, I did not see the .eln files being put in subdirectories like that
> in the Docker image.  Maybe it didn't receive that update yet.
>
> This is all very exciting.  I'm imagining an update to package.el that
> would byte-compile packages immediately, as usual, then start
> native-compile-async'ing them in the background, loading each file's
> native version as it completes, with the packages being immediately
> usable and seeming to get faster as each file is compiled and loaded.
>
> > I'm generally very satisfied (surprised) about the stability of the
> > toy. I'm looking forward going into compile time mitigation.  I guess
> > I'll start this weekend with deferred compilation and GCC profiling.
>
> I think it isn't a toy anymore!  :)  Thanks very much for your work on
> this.  Exciting times for Emacs, as this feature, in one form or
> another, has been eagerly awaited for years.
>
>