AC_SEARCH_LIBS and non-cdecl calling conventions (was: pkg-config wisdom)

[Ralf Wildenhues]

Hello,

* John Calcote wrote on Mon, Oct 26, 2009 at 05:46:07PM CET:
> On 10/25/2009 8:05 AM, Matěj Týč wrote:
> >There is one big issue with AC_SEARCH_LIBS: If you use a different
> >calling convention than cdecl (like stdcall, but I don't know, they've
> >just told me), you will get unresolved symbols if you try to link
> >without a proper include file (or something like that). Which means you
> >can't rely on that macro, especially when it concerns Windows libraries.
> >This is quite catastrophic, isn't it? What else than pkg-config would be
> 
> Is not this an issue with AC_CHECK_LIB also? I don't believe
> AC_SEARCH_LIBS suffers any greater issues than AC_CHECK_LIB.

Of course both are in the same boat.

> The problem is that both macros use a common prototype for all
> symbols: char * funcname(void);
> 
> cdecl is assumed here, because no other calling convention attribute
> is supplied on the prototype.

Yep.

> Ralf, one possible enhancement might be to allow the user to specify
> the function prototype manually in an optional argument to either of
> these macros. This might very well solve the C++ problem, however,

C++ is a different beast; mangled names cannot be enumerated.  In
the most general case, it pretty much needs a AC_LINK_IFELSE written
with a source that is specific to the library and function(s) in
question.  I'm not interested in that right now, because there is
little in ways of simplifying the situation, and because the use of
AC_LINK_IFELSE is straight-forward, ignoring for a moment that most
configure.ac authors are not used to use this macro.

OTOH, with calling conventions, we might just be able to do better.
See below.

> it wouldn't really solve the C non-cdecl calling convention issue
> because calling convention decoration attributes are compiler
> specific.

Autoconf can absorb (or find out, try out among a few possibilities)
compiler-specific information, that is not a big problem.  My problem
is that I don't yet understand what semantics a possible solution should
have in order to be sane, and portable.

Here's what I'm currently thinking, and some things I'm not sure about.
Feedback appreciated.

1) A compiler can generally support several calling conventions.
configure could try to find all possible ones through test compiles.
This assumes that it is possible to enumerate them, as there are only
finitely many.  Right?  Note that this assumes that no calling
convention introduces C++-like mangling based on argument types.

2) In general, you can mix functions with one calling convention with
functions that have a different calling convention (as long as the
convention for one specific function is always the same).  As long
as the implementation headers provide the right annotated declaration,
the right convention will be used.

3) The pressing issue that Matěj complained about was when configure
fails to detect all libraries, because all of them have a different
calling convention.  Right?

4) Is it possible/likely that systems have several instances of the same
library/function with different calling convention, and we need/want to
find just one or just a subset of them?

5) If (4) former is unlikely, then a simple loop over all possible
calling conventions could be done in AC_*_LIBS/AC_CHECK_FUNC*.  Whew,
that could be expensive.

6) If OTOH (5) is likely or even possible, then would it be sensible to
- allow the developer writing configure.ac to limit the number of
  possible calling conventions,
- let the person running configure enumerate in some way the desired set
  of calling conventions?

And would such a limit be useful enough once per configure run or needed
per AC_*_LIBS/AC_CHECK_FUNC*?

IOW, I'm trying to gauge whether to go in the direction of one of
  AC_C_CALLCONV(S)?_PUSH/POP
  AC_C_ALLOW_CALLCONV(S)?
  --enable-calling-convention=...
  just find out everything on our own, magically,
  an additional optional to AC_*_LIBS/AC_CHECK_FUNC*

or some other way we haven't thought of yet.

(And no, I'm not interested in a solution that works only for that one
operating system but cannot be made to work for others.)

Thanks,
Ralf