Re: Building a cross-compiler

[Sébastien Hinderer]

Dear Zack,

Many thanks for having taken the time to write such a detailed repsonse,
I find it really helpful.

And, many thanks, too, for your work on Autoocnf and the discussion you
started recently. As a noob I can't help much but fully support your
work.

Zack Weinberg (2021/01/21 09:20 -0500):
> Let me make sure I understand exactly what you're trying to do.  You
> want to build a cross-compiler *of OCaml*, hosted on Linux, targeting
> mingw32.

That's correct. That's the thing I'm trying to achieve as a first step.
I hope that in the end I can do much more general things, but I decided
to start the experiments here.

> You already have a cross-compiler *of C* hosted on Linux and
> targeting mingw32, which should be used to compile the parts of the
> OCaml runtime that are written in C.

Correct. All the runtime is written in C, actually. To be a bit more
precise, and although it is probably irrelevent here, there are actually
two runtimes. Indeed, OCaml programs can be compiled to either bytecode
or native code and both of them need a runtime.

> > ./configure \
> >   --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
> >   --target=x86_64-w64-mingw32
>
> Assuming this is the configure script for the OCaml implementation,
> this is abstractly the correct way to invoke it to make the cross
> compiler you want.
>
> > Since this invocation does not detect the C compiler properly
>
> Can you be more specific about what you mean by this?  Which C
> compiler did it select and why was that wrong?

Well it detected the one that runs on the build system and produces code
for the host system, which is actually not incorrect, sorry for my
wording. It's just that, since we need several C compilers, our needs do
not fit but it's not to say configure did something wrong.

> > I switched to:
> >
> > ./configure \
> >   --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu \
> >   --target=x86_64-w64-mingw32 \
> >   CC=x86_64-w64-mingw32-gcc
> >
> > But that does not quite work because, since build and host system
> > types are equal, autoconf assumes we are not in cross-compiling mode
> > and thus tries to run the test programs it compiles, which is
> > actually not possible.  I assume that what's wrong here is that
> > autoconf expects CC to point to a C compiler that runs on the build
> > system and produces code that runs on the host system and my
> > definition of CC breaks this assumption.
>
> Yes, exactly.  This is the Right Thing in the common case, even when
> you are cross-compiling, because the common case is that you're not
> building a cross-compiler at all (so --target isn't being used) and,
> when you're cross-compiling, you don't *also* need a native compiler.

Agreed.

> ax_prog_cc_for_build.m4 handles the second most common case, where
> cross compilations need a native compiler as well as a cross-compiler:
> for instance, because there are compiled-code programs that need to
> run as part of the build.  (It has some problems, but they're mostly
> problems for us Autoconf maintainers, not for you. ;-)
>
> Your situation is even more unusual: you are building a cross-compiler
> for language A, and you need a cross-compiler for language B in order
> to do so.  I'm not aware of any off-the-shelf macro libraries that
> handle this, just because building cross-compilers is itself quite
> unusual.

Many thanks for having summarized the situation, it's actually very
helpful and your summary is totally accurate.

> Most of the existing support for --target in Autoconf was
> tailored to the needs of GCC, and GCC handles this particular problem
> by *including* a C compiler -- if you configure GCC with
> --target=x86_64-w64-mingw32 --enable-languages=fortran (for instance)
> it will insist on also building the C cross-compiler that it will need
> to compile the Fortran runtime.

I understood this paragraph but am not sure what you mean here and how
to interprete it in the ocntext of the discussion?

> > So, should I perhaps introduce a second variable to designate a
> > compiler that runs on the build system but produces code for the
> > target system, with a name like TARGETCC?
>
> Yes, I think that's right.  I'd recommend you be consistent with the
> naming convention used by ax_prog_cc_for_build.m4 and call the
> variable CC_FOR_TARGET.  You should automatically deduce a default
> value for this variable from the --target option, in the same way that
> AC_CHECK_TOOL uses the value of the --host option.

Actually I got an other suggestion here that I am contemplating very
seriously.

Basically, we are on the build system and we build two kind of objects:
most of them will run on the host system, but a few of them (the
runtimes) will run on the target system. The idea is to make sure the
distinction between these two categories of objects is done in the build
system and then the part that runs on the target system could be
configured with the target being the host system.

So to speak, there would thus be the main configure invocation, done as
specified in the original mail, and this main configure would call a
sub-configure but with --host set to the target system, and that way the
usual machinery would work, it seems.

> You are also probably going to need to change all of your existing
> compilation checks -- which I presume are intended to probe
> characteristics of the *target* C runtime?

Mainly yes, I think.

> -- to use CC_FOR_TARGET instead of CC.

It seems to me that's the kind of problem that owuld be solved by the
sub-configure trick, no?

> This is going to be awkward.  We would be glad to work
> with you on improvements to Autoconf's infrastructure to make this job
> easier.

That would be an amazing outcome of this job! I don't know exactly how
things could be improved, I don't even know whether there is something
that could be improved but I am very interested in working on it!

> And since you say autoconf "tries to run the test programs it
> compiles", that suggests you're using AC_RUN_IFELSE, which just plain
> doesn't work when cross-compiling;

We indeed have one explicit invocation of this macro, to probe for
features of the mmap system call. But the iinvocation that failed was
not done by our build system, I think. As far as I remember it may have
been done by libtool. Actually, libtool is something we may want to get
rid of, because we do not actually use it  to build our libraries, so
it's there just because there were a few things libtool was able to
determine but not autoconf (I can dig to figure out exactly what, if
someone is interested).

Anyway, as explained earlier, I know autoocnf has ways to disable runing
test when in cross-compile mode, but here it didn't enter cross-compile
mode. If we had a sub-configure for the runtime, then we would git the
name of the target system as arguemnt of --host and then we would enter
cross-compile mode and, if I understand correctly, the tests would not
be run.

> you will also need to find a way to
> probe the target C runtime *without* running test programs.  For
> inspiration I suggest looking at _AC_COMPUTE_INT_COMPILE works --
> https://git.savannah.gnu.org/cgit/autoconf.git/tree/lib/autoconf/general.m4#n3274
> .

I will definitely have a look, thanks. But I'm guessing that it will
simply not be possible to figure out all the properties of the target
system and that the only left alternative will be ot hard-code them,
right?

> > Or, since we will also need a C compiler running on host and producing
> > code for target, would it perhaps be better to introduce a HOSTCC
> > variable, which can also be used on the buildhost under the hypothesis
> > given above that build==host?
>
> I don't follow -- why do you also need a C compiler running on host
> and producing code for target?

Sorry for not having been clear. With the OCaml compiler, ocamlc, you
can run commands like

ocamlc -c foo.c

ocamlc will then call the C compiler and give it the right flags so that
your object file can then be linked with OCaml modules. And obviously,
the ocamlc we are talking about will run on the host system (and so will
the underlying c compiler it calls). And the object file produced by
calling the C coompiler will need to run on the target system with the
other OCaml modules. Am I making sense now?

Thanks again for your help,

Sébastien.