Re: AC_SEARCH_LIBS and non-cdecl calling conventions

[Ralf Wildenhues]

* John Calcote wrote on Tue, Oct 27, 2009 at 09:20:44PM CET:
> Passing a fully-specified function prototype in the macro is a
> reasonable thing to do for the sake of various C++ mangling schemes,
> as well as for the sake of other calling conventions that require
> the number and/or types of parameters to determine the proper
> mangled symbol name. However, passing compiler-specific
> calling-convention attributes (e.g., __attribute(stdcall),
> __stdcall, __declspec(stdcall), __fastcall, __cdecl, etc.,) will
> clearly not work, because they're compiler-specific. Each compiler
> (even on the same system!) uses different such keywords to specify
> the same symbol attributes.

Please take a step back, and try to forget the idea of the configure.ac
author passing in a fully specified function prototype (as would be
needed for C++ mangling) for the moment.

Please also ignore efficiency issues completely for the moment.  All of
these issues are orthogonal to what I am after, and can be discussed
afterwards, when they are not premature optimization.

Let's think with the Autoconf implementor hat on.  We can change macros.
We can let AC_CHECK_LIB tests use any number of different attributes,
attribute notation syntaxes of different compilers, and calling
conventions denoted by those attributes, *as* *long* *as* each of these
are only finitely many, or can be delimited to finitely many by some
other decisive tests.

In the current scheme, AC_CHECK_LIB only tries to *link* a function
call, not to actually run the linked code.  There are two aspects here
to consider:

1) This might not be a sharp enough test to decide which calling
convention is used (i.e., linking may succeed, but running would fail).
2) It might depend upon the function whether a link with the currently
tested calling convention works: it might succeed with f1 but fail with
f2.

Now, I'm not even interested in (1) right now; and I am aware that
solving (1) in general is probably a tough problem.  All I want to know
is: is (2) not a problem, and is there only a finite set of name
mangling operations done by ways of C calling convention per compiler,
unlike is done for C++?

If the only exception to this is the number of arguments that is encoded
in the mangled name, then that might still be workable, but then it will
definitely not be cheap in the end.  Everything else can probably be
made pretty cheap, at least if the calling convention is the same for
all AC_CHECK_LIB* tests.

Thanks,
Ralf