Re: MSYS+MSVC for libtool branch-2-0, take 2

[Ralf Wildenhues]

Hi Peter,

To prevent another week long latency, I'll do a couple of replies,
starting with the easy issues.

* Peter Ekberg wrote on Tue, Jul 05, 2005 at 11:01:37PM CEST:
> Ralf Wildenhues wrote:
> > 
> > I'm very sorry it took me so long to write this email.

> Besides, I'm taking revenge by sending a mail that I've been
> writing on for about three weeks, so it's probably long and
> winding and twisted in various ways :-)

I knew you'd do that.  :)

> > * Peter Ekberg wrote on Fri, Jun 17, 2005 at 03:15:33PM CEST:
> > > Ralf Wildenhues wrote:
> > > > * Peter Ekberg wrote on Fri, Jun 10, 2005 at 10:57:33PM CEST:
> > > > > Ralf Wildenhues wrote:
> > > "$AR x archive" has to be hidden in some construct, as there is
> > > the problem that there is no way to extract all members from
> > > a MSVC .lib in one go, you have to loop over the archived
> > > files and extract them one by one.
> > 
> > We have more than one variable which allows a set of commands.
> > extract_old_archive_cmds comes to mind as a new name; and in fact, I
> > have suggested this before in another mail in this thread.
> > Seeded with '$AR x $oldlib'.  Would you like me to show this 
> > in a patch?
> 
> Ah, I must have missed that. I'll look into changing the old archive
> extraction part of the patch if you set up the framework for me.
> So a patch will indeed be helpful, I will probably have a hard time
> getting it right and will probably wait for (more) hours while
> bootstrap is running in the background... :-/

To speed up bootstrap somewhat while you are testing:
  reconfdirs=. ./bootstrap

skips all the test directories.

> > (Don't worry too much about global variable names in shell functions.
> > This is orthogonal, and I have an idea to hide the resulting 
> > ugliness in ltmain.m4sh.)
> 
> This I don't understand, so I'm not worrying, ok? :-)

OK.

> > Alternatively, if you don't like the extract_old_archive_cmds 
> > idea, I'd
> > still prefer something like
> >   $AR $AR_FLAGS$AR_SEP$oldlib ..
> > over your solution.  Have AR_SEP=' ' by default.
> > Rationale: Less variables to keep track of.
> 
> Hmmm, Automake (?) seems to set up a rule to use $AR, but
> that rule uses "$AR $ARFLAGS foo..." w/o underscore in
> AR_FLAGS (and of course no $AR_SEP).
> 
> I noticed this when I added AR_FLAGS and AR_SEP with AC_SUBST
> in an attempt to pass one of the failed tests. (I added AR_SEP
> to AC_SUBST thinking that Automake would catch up and make use
> of it in the future...)

Exactly.  And it'd be backward-compatible with what users expect, too.
Hmm.  Except I don't know how to set a Make variable to SPACE.  Maybe
Automake could use $AR $ARFLAGS$${AR_SEP-' '} or so?  Oh, chucks, we
should ask Automake for their preferred way before we set ours in
stone.  Will write there.  The only thing I could come up so far is
this heavyly ugly rule:

        AR_SEP="$(AR_SEP)"; if test -z "$$AR_SEP"; then AR_SEP=' '; fi; \
        $(AR) $(ARFLAGS)$${AR_SEP}$(OLDLIB) ...

AC_SUBSTing AR_SEP from Libtool's side is probably wrong, if it's used
in Automake.  Automake need not use Libtool..

> > By looking at `lib -link' I believe issue 1. is actually the most
> > problematic, since `lib' encodes the path into the archive by default.
> > I have not found a switch to turn this off (which does not imply there
> > is none).  But that means we have to symlink/copy objects before
> > archiving them.  (Could be done in the same loop as the renaming, but
> > will be expensive anyway.)
> > 
> > Otherwise, all use of convenience archives breaks (and running the
> > Libtool test suite really shows that).
> 
> If I do
> $ lib -OUT:foo.lib a/b/c.obj
> $ lib -EXTRACT:a/b/c.obj foo.lib
> I get a fresh copy of c.obj in cwd
> 
> If I do
> $ ar cru bar.lib a/b/d.obj
> $ ar x bar.lib
> I get a fresh copy of d.obj in cwd
> 
> Where's the difference?

Well, first problem is that POSIX specifies that no paths be encoded
into archives at all.  I don't like to violate this.
(Last but not least being that I would like that installed .lib files
be handled correctly by, e.g., cygwin ar.  Maybe that is already the
case.)

I need to test this, though, I vaguely remember members being extracted
into subdirs.

> > > (It turns out that both ar and lib works as the archiver
> > >  since MSVC does not actually see the generated archives.
> > >  ar is selected by default, so you have to specify AR=lib
> > >  or AR="link -lib" to get the Microsoft tool.)
> > 
> > The "encodes paths" is one reason to prefer `ar' over `lib' any time.
> 
> See above, and some tests that pass with lib seem to fail when
> ar is used. Haven't looked closely though.

OK.

> > > > > > After working around these, the next failure is when 
> > > > linking libltdl:
> > > > > > 
> > > > > > | libltdl_la-ltdl.obj : error LNK2019: unresolved external 
> > > > > > symbol _lt_libltdl_LTX_preloaded_symbols referenced in 
> > > > > > function _lt_dlinit.
> > > > > > | .libs/ltdl-6.dll : fatal error LNK1120: 1 
> > unresolved externals
> > > > > 
> > > > > Haven't tried libltdl, I do not depend on it for the project
> > > > > I'm targeting and I have a hard time bootstrapping libtool
> > > > > so I haven't tried the testsuite myself.

This seems to be solved, looking at your test results.  Cool!

> > > > > > The `mv' command in this line
> > > > > > | +      _LT_TAGVAR(archive_cmds, $1)='$CC -o $lib $libobjs 
> > > > > > $compiler_flags `$ECHO "X$deplibs" | $Xsed -e '\''s/ 
> > > > > > -lc$//'\''` -link -dll~linknames=~mv .libs/${libname}`$ECHO 
> > > > > > "X${release}" | $Xsed -e +s/[.]/-/g`${versuffix}.lib 
> > > > > > .libs/${libname}.lib'
> > > > > > 
> > > > > > is also broken w.r.t lib name.
> > > > > 
> > > > > I don't see what you mean here. It does what I intend it to do,
> > > > > it renames the import lib to not have a version number. Why
> > > > > is that a problem? It matches the content of the .la file.
> > > > 
> > > > I didn't keep a note on this due to time constraints, 
> > sorry.  Was a
> > > > problem with the paths again, I think (i.e., correct quoting).
> > > 
> > > Well, I didn't even try to solve problems caused by spaces in paths,
> > > I used my little -L/msvc/lib workaround instead.
> > 
> > OK.
> 
> I no longer need that with this patch, it scans the LIB variable.

Cool.

> > > > > - Add paths given with -L to the LIB variable so that the linker
> > > > >   actually sees them. This should be done after converting them
> > > > >   to win32 paths, which is easy on Cygwin and harder on MSYS.
> > 
> > > Maybe not so hard on MSYS after all, how about:
> > > win32=$(cmd \\/C echo "$unix " | sed -e "s/\"\(.*\) \" /\1/")
> > > 
> > > (cmd is not directly CMD.EXE, but a shell script provided by MSYS
> > > that invokes $COMSPEC. One question is how portable /C is as an
> > > option to $COMSPEC? I would guess that /C works for all command
> > > interpreters from Microsoft, but I'm unsure about others...)
> > 
> > This /might/ be a question for the msys/mingw mailing lists.
> > What is COMSPEC, by the way?
> 
> COMSPEC is an env variable that is the full path of the windows
> command line interpreter, e.g. C:\WINDOWS\command.com or
> C:\WINXP\System32\cmd.exe.

OK.

> > The `echo' might be problematic, if any of the shells' builtin and/or
> > external echo commands interpret backslashes.  The MinGW 
> > shell should be
> > fine with `printf %s ".."', I believe.
> 
> That echo will be fed to the builtin echo in the windows command
> line interpreter, so it should not be a problem. Or where you
> referring to something else?
> 
> The unknown is what happens if there are some non-MS command
> line interpreter installed. I remember using DRDOS or something
> ~10 years ago, and don't have a clue if those alternative command
> line interpreters are still in use or if they even work in a
> Windows environment. I'm willing to take a slight risk and
> assume that they are compatible in this regard...

I'm rather worried about a possible cross-compiling case.  Not that it's
likely someone would actually do that (although with wine or similar, it
might actually be possible already).  Then your `echo' is not guaranteed
to be fine.  (It's rather that I'd generally like to avoid anything that
hampers cross-compiling, so at least the number of issues with it does
not increase.)

> > Also I don't like the fact that we do not reuse 
> > fix_srcfile_path.  This
> > should set the right thing for cygwin/mingw and we should use 
> > it in the
> > appropriate places.  Anything more than a
> >   test -z "$fix_srcfile_path" \
> >       && $cmd
> 
> The problem is that fix_srcfile_path is not always the best thing
> to do for MSYS. It must be used in sooo many more places on Cygwin
> than on MSYS. MSYS != Cygwin, contrary to recent posts on this
> list, this is one area where their design goals differ vastly.
> 
> So, if this functionality is moved to a func/var, there still has
> to be some other logic involved than a simple flag that says yes/no
> on whether to invoke the functionality.

Well, but that should not be a problem, should it?

> > should IMHO be separated out into a function, in order to keep clutter
> > out of the main (and IMVHO already too large) functions.
> 
> I don't see an obvious solution, so I'll leave it for now.

OK.  I'll try to work on it soon.

> > > PS. With msys_msvc-4.patch I configure with
> > > ./configure CC="cl -MD" AR=lib LDFLAGS=-L/msvc/lib
> > > 
> > > I also have this line in my msys /etc/fstab
> > > C:/PROGRA~1/MICROS~4/VC98 /msvc
> > 
> > Ahh.  Guess that might be a good idea to put in corresponding
> > documentation.
> No longer needed with this patch. I'm feeding the paths in LIB
> to sys_lib_search_path_spec. Those paths can't have spaces in
> them, so that's what should go into the docs I suppose...

OK.

> > Peter, I'm sorry to sound so picky at times.  If you'd like 
> > me to, I can
> > eventually do some work on this as well, but it will be a little more
> > while at least.  Thanks again for your work.
> 
> I have full understanding and realize that my patches are more
> than likely to not conform with various criteria. If I fail to
> change anything, I either did not understand what you meant or
> did not know how to change it. Just keep flinging it my way, I
> can take it. If there were no complaints I'd assume that I was
> being ignored, it feels as if I'm just gussing most of the
> time...

OK.  You are doing just fine.  :)

> > | -   $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> > | +   if test "$with_gcc" = "yes"; then
> > | +     $run $LTCC $LTCFLAGS -s -o $cwrapper $cwrappersource
> > | +   else
> > | +     $run $LTCC $LTCFLAGS -o $cwrapper $cwrappersource
> > 
> > Why is this necessary?
> 
> Because MSVC does not have a -s option. It's not strictly
> necessary since a warning is the only evidence, but I feel
> uncomfortable with unrecognized options...

Ah, ok.  Let's solve this differently..

> Besides, I think MSVC creates "release" binaries unless asked
> to do differently. Not sure if they can be "stripped" further
> though...

and let's solve this, too.

> New issues:
> 
> In the project where I first developed the patch, functions
> and variables are marked with the appropriate
> __declspec(dllimport) and __declspec(dllexport), so exporting
> and importing variables for shared libs failed miserably for
> the general case for patches up to an including -4. In -5 this
> is fixed. But I think I've hit a bit of a snag, importing a
> variable from a dll requires that the "data item in the client
> program must be declared using extern __declspec(dllimport)".
> The snag is that it isn't known at compile time what the
> compiled unit will be used for, so this has to be solved in
> the project using libtool, if a clean solution is desired.

This sounds just like the issue Howard Chu has described on one of the
libtool lists (I believe this one) a couple of weeks ago.
I believe the bottom line is: don't do data exports, only functions
exports.  (libltdl suffers badly from this, too.)

> I found this out when fixing problems found by the tests. This
> version of the patch exports functions and variables correctly.
> It also imports function correctly, but importing variables are
> cludgy. I always say extern __declspec(dllimport), even if
> linking statically or exporting. This works, but results in
> various linker warnings. At least it works if I use lib (or
> "link -lib") as archiver. There is some failure when using ar
> as archiver that I haven't investigated, so I'm avoiding that
> ATM...

Hmm.  I am pretty sure this will fail in various special cases.
Cannot tell ATM, though.

> How do I get configure to not find g77 when it looks for a
> Fortran compiler?

g77=:
?

> Version -5 of the patch has problems exporting symbols from C++,
> I need to adjust to MSVC style symbol mangling, I think, but
> I'm not sure what's really going on.
> 
> Also missing is support for NM=dumpbin, which needs something
> along the lines of what is done for AR=lib, methinks.

Ah, ok.

> The changes to libltdl/Makefile.am needs to be done properly.

> My environment:
> 
> I have killed /etc/config.site from the MSYS/MinGW install.

The maintainer has agreed to remove this.

> I have made sure that the paths in the LIB variable for MSVC
> does not contain any spaces (uses PROGRA~1 etc).
> 
> I configure with
> ./configure CC=cl CFLAGS=-MD CXX=cl CXXFLAGS=-MD LD=link AR=lib STRIP=:
> 
> Here is an annotated dump of make check results:

It'd be nice to see the output of
  make check VERBOSE=X
(it'd suffice for the test groups in which failures occur.  But don't
waste time trying to sort these out if you don't want to :)

> PASS: tests/link.test
> PASS: tests/link-2.test
> PASS: tests/nomode.test
> PASS: tests/objectlist.test
> PASS: tests/quote.test
> FAIL: tests/sh.test
>       This is due to the use of `...` instead of $(...)
>       in func_msvc_dashL_to_LINK when assigning to LINK.
>       I'm not fluent enough in shell programming to know
>       how to properly quote to get a working version
>       that does not fail this test...

Easy to fix.

> PASS: tests/suffix.test
> SKIP: tests/tagtrace.test
> PASS: tests/cdemo-static.test
> PASS: tests/cdemo-make.test
> PASS: tests/cdemo-exec.test
> PASS: tests/demo-static.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/demo-inst.test
> PASS: tests/demo-unst.test
> PASS: tests/depdemo-static.test
> PASS: tests/depdemo-make.test
> PASS: tests/depdemo-exec.test
> PASS: tests/depdemo-inst.test
> PASS: tests/depdemo-unst.test
> PASS: tests/mdemo-static.test
> PASS: tests/mdemo-make.test
> PASS: tests/mdemo-exec.test
> PASS: tests/mdemo-inst.test
> PASS: tests/mdemo-unst.test
> PASS: tests/cdemo-conf.test
> PASS: tests/cdemo-make.test
> PASS: tests/cdemo-exec.test
> PASS: tests/demo-conf.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/demo-inst.test
> PASS: tests/demo-unst.test
> FAIL: tests/demo-deplibs.test
>       Rule for $AR by automake? See comments above...
> PASS: tests/depdemo-conf.test
> PASS: tests/depdemo-make.test
> PASS: tests/depdemo-exec.test
> PASS: tests/depdemo-inst.test
> PASS: tests/depdemo-unst.test
> PASS: tests/mdemo-conf.test
> PASS: tests/mdemo-make.test
> PASS: tests/mdemo-exec.test (runs .lib?)
> PASS: tests/mdemo-inst.test (runs .lib?)
>       Even though the above two tests pass, a dialog pops up
>       with a failure, haven't looket further...
> PASS: tests/mdemo-unst.test
> PASS: tests/mdemo-dryrun.test
> PASS: tests/mdemo2-conf.test
> PASS: tests/mdemo2-make.test
> PASS: tests/mdemo2-exec.test
> PASS: tests/pdemo-conf.test
> FAIL: tests/pdemo-make.test
>       link.exe does not behave as ld...
>       Probably need to teach libtool to use shorter command
>       lines, right?

I don't understand this comment?

> SKIP: tests/pdemo-exec.test
> SKIP: tests/pdemo-inst.test
> PASS: tests/demo-nofast.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/demo-inst.test
> SKIP: tests/demo-unst.test
> PASS: tests/depdemo-nofast.test
> PASS: tests/depdemo-make.test
> PASS: tests/depdemo-exec.test
> PASS: tests/depdemo-inst.test
> SKIP: tests/depdemo-unst.test
> PASS: tests/demo-pic.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/demo-nopic.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/cdemo-shared.test
> PASS: tests/cdemo-make.test
> PASS: tests/cdemo-exec.test
> PASS: tests/demo-shared.test
> PASS: tests/demo-make.test
> PASS: tests/demo-exec.test
> PASS: tests/demo-inst.test
> FAIL: tests/demo-hardcode.test
>       Rule for $CC by automake?

Probably.

> FAIL: tests/demo-relink.test
>       foo not found in hello-2.dll.
> PASS: tests/demo-noinst-link.test
> PASS: tests/demo-unst.test
> PASS: tests/depdemo-shared.test
> PASS: tests/depdemo-make.test
> PASS: tests/depdemo-exec.test
> PASS: tests/depdemo-inst.test
> FAIL: tests/depdemo-relink.test
>       func_l3 not found in l3-0.dll.
> PASS: tests/depdemo-unst.test
> PASS: tests/mdemo-shared.test
> PASS: tests/mdemo-make.test
> PASS: tests/mdemo-exec.test (runs .lib?)
> PASS: tests/mdemo-inst.test (runs .lib?)
>       Even though the above two tests pass, a dialog pops up
>       with a failure, haven't looket further...
> PASS: tests/mdemo-unst.test
> PASS: tests/cdemo-undef.test
> PASS: tests/cdemo-make.test
> PASS: tests/cdemo-exec.test
> PASS: tests/duplicate_members.test
> PASS: tests/tagdemo-static.test
> PASS: tests/tagdemo-make.test
> PASS: tests/tagdemo-exec.test
> PASS: tests/tagdemo-conf.test
> FAIL: tests/tagdemo-make.test
>       Does not build shared libs?
>       Problem with MSVC style C++ symbol mangling?

I'd need to see the exact failure to be able to tell.

> SKIP: tests/tagdemo-exec.test
> PASS: tests/tagdemo-shared.test
> FAIL: tests/tagdemo-make.test
> SKIP: tests/tagdemo-exec.test
> PASS: tests/tagdemo-undef.test
> PASS: tests/tagdemo-make.test
> PASS: tests/tagdemo-exec.test
> FAIL: tests/f77demo-static.test
>       g77 probably does not work with lib and other
>       Microsoft tools.

ACK.  Don't worry about this at all.

> SKIP: tests/f77demo-make.test
> SKIP: tests/f77demo-exec.test
> FAIL: tests/f77demo-conf.test
> SKIP: tests/f77demo-make.test
> SKIP: tests/f77demo-exec.test
> FAIL: tests/f77demo-shared.test
> SKIP: tests/f77demo-make.test
> SKIP: tests/f77demo-exec.test
> ====================================
> 11 of 103 tests failed
> (13 tests were not run)
> Please report to address@hidden
> ====================================

I'll comment on your patch in another mail.  Just another question: did
you try msys/gcc or cygwin/gcc with the changes to the test suite
applied?

Cheers,
Ralf