RE: MSYS+MSVC for libtool branch-2-0, take 2

[Peter Ekberg]

Ralf Wildenhues wrote:
> Hi Peter,
> 
> I'm very sorry it took me so long to write this email.
> Nice to see you are making progress.  :)

Doing my best, it's not always so easy to decypher what's
going on in guts of libtool...

Besides, I'm taking revenge by sending a mail that I've been
writing on for about three weeks, so it's probably long and
winding and twisted in various ways :-)

> * Peter Ekberg wrote on Fri, Jun 17, 2005 at 03:15:33PM CEST:
> > Ralf Wildenhues wrote:
> > > * Peter Ekberg wrote on Fri, Jun 10, 2005 at 10:57:33PM CEST:
> > > > Ralf Wildenhues wrote:
> > > > > 
> > > > > `link -LIB' can be a problem if the cygwin `link' (to create 
> > > > > hard links)
> > > > > comes before the win32 paths.  `LIB' works here.
> > > > 
> > > > I have a fix for that in the attached patch, where you can
> > > > override by setting MSLINK to the appropriate executable.
> > > 
> > > Hmm.  Can't we just move all this stuff to the section in 
> libtool.m4
> > > where $AR is set?  It already allows user override (by setting AR
> > > appropriately during configure).  We could test whether `ar' 
> > > works, and
> > > if not, whether either one of `link -lib' or `lib' work.
> > > (This is where we then should also test whether `ar' 
> > > understands the `S'
> > > option, and adjust archive_cmds/postinstall_cmds accordingly, 
> > > but that's
> > > totally unrelated to this patch.)
> > 
> > Of course, reuse the AR variable. *smacks forehead*
> 
>   :)
> 
> > Looking closer reveals a couple of problems though...
> > $AR is used in three ways if I read it correctly:
> > 1. "$AR $AR_FLAGS archive files..." to create archives.
> > 2. "$AR x archive" to extract archives.
> > 3. "$AR t archive" to list archive content.
> > 
> > I see no way to set AR and AR_FLAGS so that 1. is expanded
> > to "lib -OUT:archive files..." as is needed here (-OUT:
> > has to be prepended to the archive name without space).
> 
> Hmm.  See below.
> 
> > "$AR x archive" has to be hidden in some construct, as there is
> > the problem that there is no way to extract all members from
> > a MSVC .lib in one go, you have to loop over the archived
> > files and extract them one by one.
> 
> We have more than one variable which allows a set of commands.
> extract_old_archive_cmds comes to mind as a new name; and in fact, I
> have suggested this before in another mail in this thread.
> Seeded with '$AR x $oldlib'.  Would you like me to show this 
> in a patch?

Ah, I must have missed that. I'll look into changing the old archive
extraction part of the patch if you set up the framework for me.
So a patch will indeed be helpful, I will probably have a hard time
getting it right and will probably wait for (more) hours while
bootstrap is running in the background... :-/

> (Don't worry too much about global variable names in shell functions.
> This is orthogonal, and I have an idea to hide the resulting 
> ugliness in ltmain.m4sh.)

This I don't understand, so I'm not worrying, ok? :-)

> Alternatively, if you don't like the extract_old_archive_cmds 
> idea, I'd
> still prefer something like
>   $AR $AR_FLAGS$AR_SEP$oldlib ..
> over your solution.  Have AR_SEP=' ' by default.
> Rationale: Less variables to keep track of.

Hmmm, Automake (?) seems to set up a rule to use $AR, but
that rule uses "$AR $ARFLAGS foo..." w/o underscore in
AR_FLAGS (and of course no $AR_SEP).

I noticed this when I added AR_FLAGS and AR_SEP with AC_SUBST
in an attempt to pass one of the failed tests. (I added AR_SEP
to AC_SUBST thinking that Automake would catch up and make use
of it in the future...)

> > 3 is easy to solve, just introduce $AR_LFLAGS and set it to
> > "t" by default and to "-NOLOGO -LIST" for MSVC.
> > 
> > I have made an attempt to solve these issues in
> > msys_msvc-4.patch.
> 
> By looking at `lib -link' I believe issue 1. is actually the most
> problematic, since `lib' encodes the path into the archive by default.
> I have not found a switch to turn this off (which does not imply there
> is none).  But that means we have to symlink/copy objects before
> archiving them.  (Could be done in the same loop as the renaming, but
> will be expensive anyway.)
> 
> Otherwise, all use of convenience archives breaks (and running the
> Libtool test suite really shows that).

If I do
$ lib -OUT:foo.lib a/b/c.obj
$ lib -EXTRACT:a/b/c.obj foo.lib
I get a fresh copy of c.obj in cwd

If I do
$ ar cru bar.lib a/b/d.obj
$ ar x bar.lib
I get a fresh copy of d.obj in cwd

Where's the difference?

> > (It turns out that both ar and lib works as the archiver
> >  since MSVC does not actually see the generated archives.
> >  ar is selected by default, so you have to specify AR=lib
> >  or AR="link -lib" to get the Microsoft tool.)
> 
> The "encodes paths" is one reason to prefer `ar' over `lib' any time.

See above, and some tests that pass with lib seem to fail when
ar is used. Haven't looked closely though.

> > > > > After working around these, the next failure is when 
> > > linking libltdl:
> > > > > 
> > > > > | libltdl_la-ltdl.obj : error LNK2019: unresolved external 
> > > > > symbol _lt_libltdl_LTX_preloaded_symbols referenced in 
> > > > > function _lt_dlinit.
> > > > > | .libs/ltdl-6.dll : fatal error LNK1120: 1 
> unresolved externals
> > > > 
> > > > Haven't tried libltdl, I do not depend on it for the project
> > > > I'm targeting and I have a hard time bootstrapping libtool
> > > > so I haven't tried the testsuite myself.
> > > 
> > > Is that only because you'd need recent Autoconf/Automake 
> (from CVS)?
> > 
> > Yes. That, and I'm unsure of if there are any patches needed for
> > Cygwin (which I guess I would be using to bootstrap) and since
> > Cygwin has a strange side-by-side installation of autotools.
> 
> I vaguely remember that you need to have only one of the packages
> installed (e.g., only Autoconf-2.59, but not 2.13).

I got autoconf-cvs running and can now bootstrap

> > No, I tried bootstrapping with Cygwin and ac 2.59 and it of
> > course didn't work...
> 
> Copy and paste output?

Can't remember, no longer a problem for me so I'm not wasting time
on it... :-P

> > > > > The `mv' command in this line
> > > > > | +      _LT_TAGVAR(archive_cmds, $1)='$CC -o $lib $libobjs 
> > > > > $compiler_flags `$ECHO "X$deplibs" | $Xsed -e '\''s/ 
> > > > > -lc$//'\''` -link -dll~linknames=~mv .libs/${libname}`$ECHO 
> > > > > "X${release}" | $Xsed -e +s/[.]/-/g`${versuffix}.lib 
> > > > > .libs/${libname}.lib'
> > > > > 
> > > > > is also broken w.r.t lib name.
> > > > 
> > > > I don't see what you mean here. It does what I intend it to do,
> > > > it renames the import lib to not have a version number. Why
> > > > is that a problem? It matches the content of the .la file.
> > > 
> > > I didn't keep a note on this due to time constraints, 
> sorry.  Was a
> > > problem with the paths again, I think (i.e., correct quoting).
> > 
> > Well, I didn't even try to solve problems caused by spaces in paths,
> > I used my little -L/msvc/lib workaround instead.
> 
> OK.

I no longer need that with this patch, it scans the LIB variable.

> > > > - Add paths given with -L to the LIB variable so that the linker
> > > >   actually sees them. This should be done after converting them
> > > >   to win32 paths, which is easy on Cygwin and harder on MSYS.
> 
> > Maybe not so hard on MSYS after all, how about:
> > win32=$(cmd \\/C echo "$unix " | sed -e "s/\"\(.*\) \" /\1/")
> > 
> > (cmd is not directly CMD.EXE, but a shell script provided by MSYS
> > that invokes $COMSPEC. One question is how portable /C is as an
> > option to $COMSPEC? I would guess that /C works for all command
> > interpreters from Microsoft, but I'm unsure about others...)
> 
> This /might/ be a question for the msys/mingw mailing lists.
> What is COMSPEC, by the way?

COMSPEC is an env variable that is the full path of the windows
command line interpreter, e.g. C:\WINDOWS\command.com or
C:\WINXP\System32\cmd.exe.

> The `echo' might be problematic, if any of the shells' builtin and/or
> external echo commands interpret backslashes.  The MinGW 
> shell should be
> fine with `printf %s ".."', I believe.

That echo will be fed to the builtin echo in the windows command
line interpreter, so it should not be a problem. Or where you
referring to something else?

The unknown is what happens if there are some non-MS command
line interpreter installed. I remember using DRDOS or something
~10 years ago, and don't have a clue if those alternative command
line interpreters are still in use or if they even work in a
Windows environment. I'm willing to take a slight risk and
assume that they are compatible in this regard...

> Also I don't like the fact that we do not reuse 
> fix_srcfile_path.  This
> should set the right thing for cygwin/mingw and we should use 
> it in the
> appropriate places.  Anything more than a
>   test -z "$fix_srcfile_path" \
>       && $cmd

The problem is that fix_srcfile_path is not always the best thing
to do for MSYS. It must be used in sooo many more places on Cygwin
than on MSYS. MSYS != Cygwin, contrary to recent posts on this
list, this is one area where their design goals differ vastly.

So, if this functionality is moved to a func/var, there still has
to be some other logic involved than a simple flag that says yes/no
on whether to invoke the functionality.

> should IMHO be separated out into a function, in order to keep clutter
> out of the main (and IMVHO already too large) functions.

I don't see an obvious solution, so I'll leave it for now.

> > > > - libtool at some point thinks /mingw/bin/ld is the linker and
> > > >   uses that to deduce stuff, perhaps other configure checks are
> > > >   also a problematic, I haven't looked closely since it works
> > > >   for me.
> > > 
> > > Show `./libtool --config'.  In order to use only the msvc 
> tools, it
> > > might be necessary to configure somehow like
> > >   CC=cl CXX=cl LD=link AR=link\ -lib
> > > iff the respective mingw/cygwin tools are available as well. 
> > > I regard this as a feature, BTW, not a bug.
> > > 
> > > It'll likely be some time before I can look at your patch again.
> > > It's good to see that it's getting smaller, though.  :)
> > > 
> > > Another couple of hints: if you intend to work on the 
> path conversion
> > > thing, please separate that out into a shell function if at all
> > > possible.  (And have a reasonably fast default path so that 
> > > not everyone
> > > has to suffer from it.)
> > 
> > One more thing that perhaps should be handled is that the content
> > of the LIB variable should be added to sys_lib_search_path (or
> > some other variable?) This would require converting paths from
> > win32 style to unix style, which again is easy on Cygwin, but
> > much harder on MinGW (the workaround above does not work).
> > But I would not consider this a showstopper. The user can always
> > add the needed paths with -L as I have done.
> > 
> > PS. With msys_msvc-4.patch I configure with
> > ./configure CC="cl -MD" AR=lib LDFLAGS=-L/msvc/lib
> > 
> > I also have this line in my msys /etc/fstab
> > C:/PROGRA~1/MICROS~4/VC98 /msvc
> 
> Ahh.  Guess that might be a good idea to put in corresponding
> documentation.

No longer needed with this patch. I'm feeding the paths in LIB
to sys_lib_search_path_spec. Those paths can't have spaces in
them, so that's what should go into the docs I suppose...

> > PPS. The patch does not solve as many issus on Cygwin as it
> > does on MSYS. Therefore, please don't judge it by its
> > performance on Cygwin. You have to walk before you can run...
> 
> Sure.
> 
> More comments below inline.
> 
> Peter, I'm sorry to sound so picky at times.  If you'd like 
> me to, I can
> eventually do some work on this as well, but it will be a little more
> while at least.  Thanks again for your work.

I have full understanding and realize that my patches are more
than likely to not conform with various criteria. If I fail to
change anything, I either did not understand what you meant or
did not know how to change it. Just keep flinging it my way, I
can take it. If there were no complaints I'd assume that I was
being ignored, it feels as if I'm just gussing most of the
time...

> | @@ -6008,11 +6094,15 @@
> |       # we should really use a build-platform specific compiler
> |       # here, but OTOH, the wrappers (shell script and this C one)
> |       # are only useful if you want to execute the "real" binary.
> |       # Since the "real" binary is built for $host, then this
> |       # wrapper might as well be built for $host, too.
> | -     $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> | +     if test "$with_gcc" = "yes"; then
> | +       $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> | +     else
> | +       $run $LTCC $LTCFLAGS -o $cwrapper $cwrappersource
> 
> Why is this necessary?

Because MSVC does not have a -s option. It's not strictly
necessary since a warning is the only evidence, but I feel
uncomfortable with unrecognized options...

Besides, I think MSVC creates "release" binaries unless asked
to do differently. Not sure if they can be "stripped" further
though...

New issues:

In the project where I first developed the patch, functions
and variables are marked with the appropriate
__declspec(dllimport) and __declspec(dllexport), so exporting
and importing variables for shared libs failed miserably for
the general case for patches up to an including -4. In -5 this
is fixed. But I think I've hit a bit of a snag, importing a
variable from a dll requires that the "data item in the client
program must be declared using extern __declspec(dllimport)".
The snag is that it isn't known at compile time what the
compiled unit will be used for, so this has to be solved in
the project using libtool, if a clean solution is desired.

I found this out when fixing problems found by the tests. This
version of the patch exports functions and variables correctly.
It also imports function correctly, but importing variables are
cludgy. I always say extern __declspec(dllimport), even if
linking statically or exporting. This works, but results in
various linker warnings. At least it works if I use lib (or
"link -lib") as archiver. There is some failure when using ar
as archiver that I haven't investigated, so I'm avoiding that
ATM...

How do I get configure to not find g77 when it looks for a
Fortran compiler?

Version -5 of the patch has problems exporting symbols from C++,
I need to adjust to MSVC style symbol mangling, I think, but
I'm not sure what's really going on.

Also missing is support for NM=dumpbin, which needs something
along the lines of what is done for AR=lib, methinks.

The changes to libltdl/Makefile.am needs to be done properly.



My environment:

I have killed /etc/config.site from the MSYS/MinGW install.

I have made sure that the paths in the LIB variable for MSVC
does not contain any spaces (uses PROGRA~1 etc).

I configure with
./configure CC=cl CFLAGS=-MD CXX=cl CXXFLAGS=-MD LD=link AR=lib STRIP=:

Here is an annotated dump of make check results:

$ make check
Making check in .
make[1]: Entering directory `/home/peda/libtool/2-0/libtool/msvc'
make  check-TESTS
make[2]: Entering directory `/home/peda/libtool/2-0/libtool/msvc'
PASS: tests/link.test
PASS: tests/link-2.test
PASS: tests/nomode.test
PASS: tests/objectlist.test
PASS: tests/quote.test
FAIL: tests/sh.test
        This is due to the use of `...` instead of $(...)
        in func_msvc_dashL_to_LINK when assigning to LINK.
        I'm not fluent enough in shell programming to know
        how to properly quote to get a working version
        that does not fail this test...
PASS: tests/suffix.test
SKIP: tests/tagtrace.test
PASS: tests/cdemo-static.test
PASS: tests/cdemo-make.test
PASS: tests/cdemo-exec.test
PASS: tests/demo-static.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/demo-inst.test
PASS: tests/demo-unst.test
PASS: tests/depdemo-static.test
PASS: tests/depdemo-make.test
PASS: tests/depdemo-exec.test
PASS: tests/depdemo-inst.test
PASS: tests/depdemo-unst.test
PASS: tests/mdemo-static.test
PASS: tests/mdemo-make.test
PASS: tests/mdemo-exec.test
PASS: tests/mdemo-inst.test
PASS: tests/mdemo-unst.test
PASS: tests/cdemo-conf.test
PASS: tests/cdemo-make.test
PASS: tests/cdemo-exec.test
PASS: tests/demo-conf.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/demo-inst.test
PASS: tests/demo-unst.test
FAIL: tests/demo-deplibs.test
        Rule for $AR by automake? See comments above...
PASS: tests/depdemo-conf.test
PASS: tests/depdemo-make.test
PASS: tests/depdemo-exec.test
PASS: tests/depdemo-inst.test
PASS: tests/depdemo-unst.test
PASS: tests/mdemo-conf.test
PASS: tests/mdemo-make.test
PASS: tests/mdemo-exec.test (runs .lib?)
PASS: tests/mdemo-inst.test (runs .lib?)
        Even though the above two tests pass, a dialog pops up
        with a failure, haven't looket further...
PASS: tests/mdemo-unst.test
PASS: tests/mdemo-dryrun.test
PASS: tests/mdemo2-conf.test
PASS: tests/mdemo2-make.test
PASS: tests/mdemo2-exec.test
PASS: tests/pdemo-conf.test
FAIL: tests/pdemo-make.test
        link.exe does not behave as ld...
        Probably need to teach libtool to use shorter command
      lines, right?
SKIP: tests/pdemo-exec.test
SKIP: tests/pdemo-inst.test
PASS: tests/demo-nofast.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/demo-inst.test
SKIP: tests/demo-unst.test
PASS: tests/depdemo-nofast.test
PASS: tests/depdemo-make.test
PASS: tests/depdemo-exec.test
PASS: tests/depdemo-inst.test
SKIP: tests/depdemo-unst.test
PASS: tests/demo-pic.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/demo-nopic.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/cdemo-shared.test
PASS: tests/cdemo-make.test
PASS: tests/cdemo-exec.test
PASS: tests/demo-shared.test
PASS: tests/demo-make.test
PASS: tests/demo-exec.test
PASS: tests/demo-inst.test
FAIL: tests/demo-hardcode.test
        Rule for $CC by automake?
FAIL: tests/demo-relink.test
        foo not found in hello-2.dll.
PASS: tests/demo-noinst-link.test
PASS: tests/demo-unst.test
PASS: tests/depdemo-shared.test
PASS: tests/depdemo-make.test
PASS: tests/depdemo-exec.test
PASS: tests/depdemo-inst.test
FAIL: tests/depdemo-relink.test
        func_l3 not found in l3-0.dll.
PASS: tests/depdemo-unst.test
PASS: tests/mdemo-shared.test
PASS: tests/mdemo-make.test
PASS: tests/mdemo-exec.test (runs .lib?)
PASS: tests/mdemo-inst.test (runs .lib?)
        Even though the above two tests pass, a dialog pops up
        with a failure, haven't looket further...
PASS: tests/mdemo-unst.test
PASS: tests/cdemo-undef.test
PASS: tests/cdemo-make.test
PASS: tests/cdemo-exec.test
PASS: tests/duplicate_members.test
PASS: tests/tagdemo-static.test
PASS: tests/tagdemo-make.test
PASS: tests/tagdemo-exec.test
PASS: tests/tagdemo-conf.test
FAIL: tests/tagdemo-make.test
        Does not build shared libs?
        Problem with MSVC style C++ symbol mangling?
SKIP: tests/tagdemo-exec.test
PASS: tests/tagdemo-shared.test
FAIL: tests/tagdemo-make.test
SKIP: tests/tagdemo-exec.test
PASS: tests/tagdemo-undef.test
PASS: tests/tagdemo-make.test
PASS: tests/tagdemo-exec.test
FAIL: tests/f77demo-static.test
        g77 probably does not work with lib and other
        Microsoft tools.
SKIP: tests/f77demo-make.test
SKIP: tests/f77demo-exec.test
FAIL: tests/f77demo-conf.test
SKIP: tests/f77demo-make.test
SKIP: tests/f77demo-exec.test
FAIL: tests/f77demo-shared.test
SKIP: tests/f77demo-make.test
SKIP: tests/f77demo-exec.test
====================================
11 of 103 tests failed
(13 tests were not run)
Please report to address@hidden
====================================
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory `/home/peda/libtool/2-0/libtool/msvc'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/home/peda/libtool/2-0/libtool/msvc'
make: *** [check-recursive] Error 1

Cheers,
Peter

msys_msvc-5.patch
Description: msys_msvc-5.patch