Re: Lightning loops indefinitely inside jit_emit on M1 Macs

[Paulo César Pereira de Andrade]

Em qui., 25 de mar. de 2021 às 14:02, Paulo César Pereira de Andrade
<paulo.cesar.pereira.de.andrade@gmail.com> escreveu:
>
> Em dom., 21 de mar. de 2021 às 06:12, Darren Kulp <darren@kulp.ch> escreveu:
> >
> > Hello,
>
>   Hi,
>
> > I reproduced the issues on a publicly available machine.
> >
> > Since cfarm (https://cfarm.tetaneutral.net) has just today announced 
> > availability of an M1 Mac, I reproduced both my problems on that machine 
> > (gcc304.fsffrance.org). Probably a lot of people on this mailing list have 
> > accounts there already.
> >
> > curl -O http://ftp.gnu.org/gnu/lightning/lightning-2.1.3.tar.gz
> > tar xf lightning-2.1.3.tar.gz
> > cd lightning-2.1.3
> > ./configure --enable-assertions
> > make
> > DYLD_LIBRARY_PATH=$PWD/lib/.libs ./doc/.libs/rfib
> > # The above invocation exits an assertion about MAP_FAILED
> >
> > sed -i '' s/-O2// configure
> > sed -i '' 's/MAP_ANON,/MAP_JIT | &/' lib/lightning.c
> > ./configure --enable-assertions
> > make
> > DYLD_LIBRARY_PATH=$PWD/lib/.libs ./doc/.libs/rfib
> > # The above invocation exits with a bus error
>
>   It looks like it might be required to use mremap, or have some
> system wide configuration to allow the sample binaries, and the
> check/lightning test tool to execute jit.
>
>   I do not know how work the security features. Might be something
> like selinux. As long as it is not required to be root to enable
> jit execution, I should be able to fix it in the next few days using
> the cfarm host.

  Partial copy&paste of
https://lists.exim.org/lurker/message/20200715.005734.514d1adb.es.html

"""
With that said, here is what will no longer work:

* Allocating a RWX region with no special flags and attempting to write code to
it and execute it. This will fail because you will not have write permission
due to APRR on ARM chips. Will not work on Intel chips running under hardened
runtime when the code is attempted to be executed due to codesigning, but will
work otherwise.
* Allocating a RWX region with MAP_JIT and the com.apple.security.cs.allow-jit
entitlement: will fail for the same reason above on ARM. Will succeed on
x86_64, even with runtime hardening.
* Mirror mapping/remapping, at least when I tried it this does not work. YMMV.

What will work on ARM, if the memory is correctly allocated based on the
runtime hardening setting:

* Using two threads to work with the RWX region; one with the original
effective permissions (R-X); the second which runs
pthread_jit_write_protect_np(0) to change its permissions to RW- and can now
write code.
* Using one thread: interleaving calls to pthread_jit_write_protect_np when
switching between writing to the JIT region and executing it.
* Using one thread: modifying a proprietary register before you try to write
code into this region, which will disable this APRR feature altogether for the
program:

unsigned long long mask;
__asm__ volatile("mrs %0, s3_4_c15_c2_7" : "=r"(mask): :);
__asm__ volatile("msr s3_4_c15_c2_7, %0" : : "r"(mask & 0xfffffffff0ffffff) :);

This has the upside that you only have to do it once early at program startup
and never have to deal with it again, but the downside is that I don't think
Apple would particularly approve of this.
"""

  Adding MAP_JIT to mmap flags correct mmap.
  Using the simplest approach, creating the 'mask' local, variable and
adding the inline assembly in jit_init(), when run I get:

pcpa@GCC-Farm-MiniMac lightning-2.1.3 %
DYLD_LIBRARY_PATH=$PWD/lib/.libs ./doc/.libs/rfib
zsh: illegal hardware instruction  DYLD_LIBRARY_PATH=$PWD/lib/.libs
./doc/.libs/rfib

  Trying to debug it better:

pcpa@GCC-Farm-MiniMac lightning-2.1.3 %
DYLD_LIBRARY_PATH=$PWD/lib/.libs lldb ./doc/.libs/rfib
(lldb) target create "./doc/.libs/rfib"
Current executable set to '/Users/pcpa/lightning-2.1.3/doc/.libs/rfib' (arm64).
(lldb) r
error: process exited with status -1 (developer mode is not enabled on
this machine and this is a non-interactive debug session.)

  I will check a bit more, but apparently without a graphical
interface, or debug mode enabled (not sure how to get it)
will not be able to inspect the issue.
(https://www.mail-archive.com/lldb-commits@lists.llvm.org/msg39464.html)

  Not certain if I can do much more, and not certain if it did not allow
the inline asm to allow jit, or if it did not like the generated code.

  Btw, by adding this pseudo patch to rfib.c:

   fib = jit_emit();
+jit_print();
+jit_disassemble();
   jit_clear_state();

  I see this output:
$ ./rfib
L0:
L1: /* prolog */
        arg #1
        getarg_l x9 #1
         \__ movr x9 x0
    beqi L2 x9 0x0
        movr x19 x9
        movi x9 0x1
    blei L2 x19 0x2
        subi x20 x19 0x1
        subi x21 x19 0x2
        prepare
        pushargr x20
         \__ movr x0 x20
    finishi 0x0
         \__ calli L0
        retval_l x20
         \__ movr x20 x0
        prepare
        pushargr x21
         \__ movr x0 x21
    finishi 0x0
         \__ calli L0
        retval_l x9
         \__ movr x9 x0
        addr x9 x9 x20
L2:
        retr x9
         \__ movr x0 x9
         \__ live x0
         \__ ret
L3: /* epilog */
      0x7fa54bd000      stp     x29, x30, [sp,#-160]!
      0x7fa54bd004      mov     x29, sp
      0x7fa54bd008      stp     x19, x20, [sp,#16]
      0x7fa54bd00c      str     x21, [sp,#32]
      0x7fa54bd010      mov     x9, x0
      0x7fa54bd014      cbz     x9, 0x7fa54bd04c
      0x7fa54bd018      mov     x19, x9
      0x7fa54bd01c      mov     x9, #0x1                        // #1
      0x7fa54bd020      cmp     x19, #0x2
      0x7fa54bd024      b.le    0x7fa54bd04c
      0x7fa54bd028      sub     x20, x19, #0x1
      0x7fa54bd02c      sub     x21, x19, #0x2
      0x7fa54bd030      mov     x0, x20
      0x7fa54bd034      bl      0x7fa54bd000
      0x7fa54bd038      mov     x20, x0
      0x7fa54bd03c      mov     x0, x21
      0x7fa54bd040      bl      0x7fa54bd000
      0x7fa54bd044      mov     x9, x0
      0x7fa54bd048      add     x9, x9, x20
      0x7fa54bd04c      mov     x0, x9
      0x7fa54bd050      ldp     x19, x20, [sp,#16]
      0x7fa54bd054      ldr     x21, [sp,#32]
      0x7fa54bd058      ldp     x29, x30, [sp],#160
      0x7fa54bd05c      ret
fib(32) = 2178309

  I believe iOS uses the same ABI as Linux, so, it should be just
some minimal change to get it working.

> > I will continue to try when I can to debug the issue, but maybe someone who 
> > has access to cfarm and who knows lightning will be able to see what I am 
> > missing.
> >
> > Darren Kulp

Thanks!

Paulo