Re: [PATCH] [WIP] Port feature/native-comp to Windows.

[Nicolas Bértolo]

>> As I said above other architectures or compilers should work, but it may be 
>> necessary to change the code
>> that generates calls to setjmp(), since there are many ways to do it. An 
>> option would be to copy what the
>> setjmp.h header does, but I do not think it is a good idea. The proper fix 
>> would be to use autoconf to detect it
>> somehow.

> Could you elaborate why this is an issue, and what exactly are the
> details that need to be adapted to a different setjmp implementation?

> Also, did you try compiling the modified code with the 32-bit MinGW64
> compiler?

I haven't tried to compile it with the 32-bit compiler.

There are many ways to call setjmp() in Windows. It depends on the architecture,
whether the Universal CRT is used, whether SEH is enabled, the compiler version,
etc.

#define setjmp(BUF) _setjmp3((BUF), NULL)
#define setjmp(BUF) __mingw_setjmp((BUF))
#define setjmp(BUF) _setjmp((BUF), __builtin_sponentry())
#define setjmp(BUF) _setjmp((BUF), mingw_getsp())
#define setjmp(BUF) _setjmp((BUF), __builtin_frame_address (0))
#define setjmp(BUF) _setjmp((BUF), NULL)

_setjmp may be a call to function or a macro:
#define _setjmp __intrinsic_setjmp
#define _setjmp __intrinsic_setjmpex
#define _setjmp _setjmpex

This is nicely abstracted through a macro in the setjmp.h header.
See 
https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/crt/setjmp.h

In my machine (Windows 10, GCC 10.0 64 bits) the version that works is:
#define setjmp(BUF) _setjmp((BUF), __builtin_frame_address (0))
        where _setjmp is a call to the function.

libgccjit does not implement a preprocessor, so we need to create a
function call
to the proper function with the proper arguments for the system. To do this it
is necessary to know what function to call and what arguments to give it. An
option would be to copy the logic from the Mingw64 header. I don't like this for
two reasons:

- The header may change as Microsoft adds more stuff to its C runtime or
  something else is discovered through reverse engineering. This would lead to
  weird bugs when Emacs is compiled with a version of setjmp() but it generates
  calls to it in a different style.
- There may be licensing issues?

>> Another issue is that the “emacs_dir” environment variable needs to be set 
>> quite early in the initialization
>> process. I do not know enough about the Emacs internals to make the proper 
>> changes for that, so I just
>> added a dirty hack.

> Why is this a problem for the native-compile version?

The load_pdump() function calls it. I haven't found out why yet.

>> There is a remaining issue involving the environment in which emacs runs. 
>> The libgccjit likes to run in a
>> Mingw64 environment, so it can find the assembler, linker, etc.

> What do you mean by Mingw64 environment?  Do you mean the MSYS
> environment, i.e. the one that uses Bash and can run Unix shell
> scripts?  If so, why is that needed?  Compiler passes are native
> Windows programs, not MSYS programs.  Is this something special to how
> libgccjit was ported to MinGW?

> This should be fixed, but I don't think I understand enough to propose
> the way of fixing it.

> I think describing the environment and the need for having it in this
> case will be a significant first step towards resolving the problems.

I tried copying the assembler and linker (as.exe and ld.exe) into the folder
where emacs.exe lives. It is necessary to add that folder to PATH, that is the
first issue I found. Having done that was enough to make it work up to
the point where the
linker needs to find the Windows libraries.

If I remove the MSYS installation folder then it fails with these errors:

libgccjit.so: error: error invoking gcc driver

-or-

ld: cannot find dllcrt2.o: No such file or directory
ld: cannot find crtbegin.o: No such file or directory
ld: cannot find -lmingw32
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find -lmoldname
ld: cannot find -lmingwex
ld: cannot find -lmsvcrt
ld: cannot find -lpthread
ld: cannot find -ladvapi32
ld: cannot find -lshell32
ld: cannot find -luser32
ld: cannot find -lkernel32
ld: cannot find -lmingw32
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find -lmoldname
ld: cannot find -lmingwex
ld: cannot find -lmsvcrt
ld: cannot find crtend.o: No such file or directory

You are right when you say that they are native Windows programs. They don't
need a "pseudo-unix" environment like I said previously. But they need some
support files from the MSYS installation. I haven't figured out which ones yet.

>> Subject: [PATCH 4/6] Handle LISP_WORDS_ARE_POINTERS and
>>  CHECK_LISP_OBJECT_TYPE.

> Is this specific to MS-Windows?  If so, what is the MS-Windows
> specific aspects of native compilation that require this?

This is partially specific to Windows. I had trouble compiling it with the
`--enable-check-lisp-object-type` configure option, so I had to add support for
it.

One aspect that is specific to Windows is that sizeof(void*) != sizeof(long)
even if WIDE_EMACS_INT is not defined. The code assumed that sizeof(Lisp_Word)
== sizeof(long) if WIDE_EMACS_INT was not defined. I fixed this by adding many
types that represent the Lisp_* family and changing the code to use these
instead of long and long long.

>> Subject: [PATCH 5/6] Remove a layer of indirection for access to pure 
>> storage.

> Same questions here.

This one is definitely not Windows specific. There was a bug that caused
PURE_P() to be implemented incorrectly in the generated code.

It defined a variable `pure_reloc` of type `void**` that was supposed to store a
pointer to a pointer to pure storage. It was initialized to
`(EMACS_INT**)&pure`. This is expression does not take evaluates to a pointer of
type `EMACS_INT**` that points to the start of pure_storage. Since the generated
code derefereced this pointer, it was implementing PURE_P() as

bool PURE_P(void* ptr)
{
  return ((uintptr_t) ptr - (uintptr_t)pure[0]) <= PURESIZE;
}

In my tests `pure[0]` == 2.

This bug caused the native compiler to crash by calling pure_write_error(). It
is strange that this was not detected in GNU/Linux. I conjecture that all Lisp
objects are allocated in addresses higher than `pure` in GNU/Linux and that they
are higher than `pure[0]` too. I am not sure though.

This is not the case in Windows. It is possible to have Lisp objects that are
allocated below `pure`.

Nicolas

El sáb., 9 may. 2020 a las 3:08, Eli Zaretskii (<address@hidden>) escribió:
>
> > Date: Fri, 8 May 2020 16:55:59 -0300
> > From: Nicolas Bertolo <address@hidden>
> >
> > I have ported the feature/native-comp branch to Windows. I have tested my 
> > changes in Windows 10 x64 with
> > Mingw64 GCC 10.0. Other architectures or compilers should work, but it may 
> > be necessary to adjust the
> > code a little bit.
>
> Great news, thank you for working on this.
>
> > As I said above other architectures or compilers should work, but it may be 
> > necessary to change the code
> > that generates calls to setjmp(), since there are many ways to do it. An 
> > option would be to copy what the
> > setjmp.h header does, but I do not think it is a good idea. The proper fix 
> > would be to use autoconf to detect it
> > somehow.
>
> Could you elaborate why this is an issue, and what exactly are the
> details that need to be adapted to a different setjmp implementation?
>
> Also, did you try compiling the modified code with the 32-bit MinGW64
> compiler?
>
> > Another issue is that the “emacs_dir” environment variable needs to be set 
> > quite early in the initialization
> > process. I do not know enough about the Emacs internals to make the proper 
> > changes for that, so I just
> > added a dirty hack.
>
> Why is this a problem for the native-compile version?
>
> > There is a remaining issue involving the environment in which emacs runs. 
> > The libgccjit likes to run in a
> > Mingw64 environment, so it can find the assembler, linker, etc.
>
> What do you mean by Mingw64 environment?  Do you mean the MSYS
> environment, i.e. the one that uses Bash and can run Unix shell
> scripts?  If so, why is that needed?  Compiler passes are native
> Windows programs, not MSYS programs.  Is this something special to how
> libgccjit was ported to MinGW?
>
> > This is means that Emacs needs to run in a pseudo-Unix
> > environment. I don’t like this since this environment would be
> > propagated to other processes launched by Emacs, and they may not
> > like this.
>
> This should be fixed, but I don't think I understand enough to propose
> the way of fixing it.
>
> > I have thought about a simple fix to this but I haven’t implemented it yet. 
> > The Emacs process that needs to
> > run in a Mingw64 environment is actually the subprocess that performs the 
> > compilation, not the main
> > process that runs the editor. So my idea is to run this subprocess through 
> > a small script that setups the
> > environment that libgccjit expects without polluting the Emacs environment.
>
> I think describing the environment and the need for having it in this
> case will be a significant first step towards resolving the problems.
>
> > Subject: [PATCH 4/6] Handle LISP_WORDS_ARE_POINTERS and
> >  CHECK_LISP_OBJECT_TYPE.
> >
> > * src/comp.c: Introduce the Lisp_X, Lisp_Word, and Lisp_Word_tag
> > types. These types are used instead of long or long long. Use
> > emacs_int_type and emacs_uint_types where appropriate.
> > (emit_coerce): Add special logic that handles the case when
> > Lisp_Object is a struct. This is necessary for handling the
> > --enable-check-lisp-object-type configure option.
> >
> > * src/lisp.h: Since libgccjit does not support opaque unions, change
> > Lisp_X to be struct. This is done to ensure that the same types are
> > used in the same binary. It is probably unnecessary since only a
> > pointer to it is used.
>
> Is this specific to MS-Windows?  If so, what is the MS-Windows
> specific aspects of native compilation that require this?
>
> > Subject: [PATCH 5/6] Remove a layer of indirection for access to pure 
> > storage.
> >
> > * src/comp.c: Taking the address of an array is the same as casting it
> > to a pointer. Therefore, the C expression `(EMACS_INT **) &pure` is in
> > fact adding a layer of indirection that is not necessary. The fix is
> > to cast the `pure` array to a pointer and store that in a void pointer
> > that is part of the compiled shared library.
>
> Same questions here.
>
> Thanks.