Re: [PATCH] [cygwin|mingw] fix dlpreopen with --disable-static take 2

[Peter Rosin]

Den 2009-01-05 06:24 skrev Charles Wilson:
> Charles Wilson wrote:
> > Charles Wilson wrote:
> > > Peter Rosin wrote:
> > > > I'm primarily trying to determine what impact this has on my
> > > > MSVC branch...
> > Ran some experiments on the libraries shipped with the Windows SDK. The
> > attached script worked ok on most of them. After eliminating the static
> > libraries and the import libraries to '*.dll' and '*.DLL' (241
> > "successes" in all), I'm left with the following 13 odd ducks:
> >
> > These are correct, but are a reminder that import libraries exist for
> > objects other than those named foo.dll. So that's 5 more successes:
> >
> > KSProxy.Lib :x86 archive import FOR ksproxy.ax
> > bthprops.lib :x86 archive import FOR bthprops.cpl
> > irprops.lib :x86 archive import FOR irprops.cpl
> > NetSh.Lib :x86 archive import FOR NETSH.EXE
> > WinSpool.Lib :x86 archive import FOR WINSPOOL.DRV
> >
> > These are the true failures:
> >
> > For the following 6 libraries, it is the LAST archive member with a
> > .idata$6 section, not the first one, that specifies the DLL.
> >
> > GdiPlus.lib :x86 archive import FOR u.GdiplusStartup
> > MSTask.Lib :x86 archive import FOR address@hidden
> > WS2_32.Lib :x86 archive import FOR ..inet_pton
> > ksuser.lib :x86 archive import FOR ..KsCreateTopologyNode
> > shell32.lib :x86 archive import FOR =.WOWShellExecute
> > windowscodecs.lib :x86 archive import FOR q.WICSetEncoderFormat_Proxy
> >
> > This one imports symbols from multiple DLLs. One of them happens to be
> > in the last archive member, but...
> >
> > Vfw32.Lib :x86 archive import FOR -.StretchDIB
> >
> > I have no idea what this one is. objdump can't grok it:
> >
> > MAPI.Lib :MAPI.Lib: Microsoft Visual C library
> > $ objdump -f MAPI.Lib
> > objdump: MAPI.Lib: File format not recognized
> >
> > So that's 246 PASS, 8 FAIL.
>
> So, I've prepared a patch for dlltool which adds an '--identify-ms'
> flag, which modifies the behavior of the '--identify <implib>' option.
> It searches for .idata$6 instead of .idata$7, AND attempts to
> disambiguate between "the one that specifies the DLL name" and "all the
> other ones that list the symbols" by inspecting the flags.
>
> In almost all cases, "the one that specifies the DLL name" has the flag
> value 0x00000123
>    SEC_ALLOC | SEC_LOAD | SEC_DATA | SEC_HAS_CONTENTS
> The "other ones" have flag values 0x00204103
>    SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_KEEP
>
> This version of dlltool was able to operate on all of the import
> libraries in the Windows SDK, except for:
>
> MAPI.Lib: MAPI.Lib: Microsoft Visual C library
>   === again, because bfd has no idea how to parse this one
>
> Vfw32.Lib: Import library `Vfw32.Lib' specifies two or more dlls:
> `MSVFW32.dll' and `AVIFIL32.dll'
>
> And the following:
>
> WebPost.Lib: x86 archive import FOR WEBPOST.DLL
> ddao35.lib: x86 archive import FOR ddao35.dll
> ddao35d.lib: x86 archive import FOR ddao35d.dll
> ddao35u.lib: x86 archive import FOR ddao35u.dll
> ddao35ud.lib: x86 archive import FOR ddao35ud.dll
> More on these, later.
>
> Note that this dlltool /succeeded/ on
>         GdiPlus.lib
>         MSTask.Lib
>         WS2_32.Lib
>         ksuser.lib      
>         shell32.lib
>         windowscodecs.lib
> where the script in my previous post failed. Dlltool can handle the case
> where "the one that specifies the DLL name" is not first.
>
> The five new failures (where the script succeeded) are interesting. In
> each case, the "rule" above ("the one" has flag value 0x00000123, and
> the others do not) was incorrect:
>
> ================================
> $ ~/dlltool.exe --identify WebPost.Lib --identify-ms
> flags: 0x00000123       datasize: 0000000c      data: 'WEBPOST.DLL'
> 00000000  5745 4250 4f53 542e 444c 4c00           WEBPOST.DLL.
>
> flags: 0x00000123       datasize: 00000010      data: ''
> 00000000  0400 5770 4269 6e64 546f 5369 7465 4100 ..WpBindToSiteA.
>
> /home/cwilson/dlltool: Import library `WebPost.Lib' specifies two or
> more dlls: `WEBPOST.DLL' and `'
> ================================
>
> The error message is a little confusing: `WEBPOST.DLL' and `'? The
> "empty" name is because the data contains the unprintable character \004
> followed by '\0'.  Recall for symbols, the first two bytes are a
> little-endian count. So this is symbol 0x0004. I *guess* I could check
> that both offset 0 and offset 1 contain printable characters. But that's
> still just a heuristic, because a really big DLL might have
>   "01my_symbol"
> where the first two bytes are 0x30 0x31 ('01') representing symbol
> number 0x3130 or 12352.
>
> But this wierd case occurs only when the import library appears to NOT
> follow the pattern most of them do. In fact, in these five import
> libraries, ALL of the .idata$6 sections have flag 0x00000123, not just
> "the one" we want.
>
> But, what are they? Do we care?
>
> The ddao35 libraries are the Microsoft JET 3.5 DAO C++ libraries, for
> DOS-Win16 (!).  Microsoft shipped the Jet 4.0 libraries with WinME and
> W2k, and recommends against using ones older than that. Do we care that
> you won't be able to dlpreopen (or dlltool --identify) these ancient
> import libraries?)
>
> Webpost.Lib (--> webpost.dll) seems to be part of the Web Publishing
> Wizard API. I can't find any other useful information.
>
> So, if it's worth pursing, we have all the pieces to (a) teach dlltool
> how to deal with MS-style implibs, (b) create a fallback script when new
> dlltool is not available.  However, this all presupposes that your
> libtool/MSVC support is allowed to use binutils programs...otherwise,
> things get tricky. I have a 'dllname' program that can be adapted for
> this purpose, but again: can we ship/embed non-standard binary tools?
> Assume/require that they are present? I really don't know how your
> MSVC/libtool stuff is set up.

Interesting! Meanwhile, I have done some experiments on my own, as I
don't like the dependence on anything that comes with MinGW when
dealing with libtool and MSVC.

(Speaking of dependencies, I don't think the current MinGW code
 in libtool requires 'file' to be present, and I don't think it is
 part of a minimal MSYS install. It's not in my install anyway.)

I have found that for MSVC import libraries the simplest thing is
to list the archive members to get to the dll name. I have tried
with:
  lib -nologo -list $implib | grep -v '\.obj$' | sort -u
or, in gnu speak:
  ar t $implib | grep -v '\.obj$' | sort -u

This works for all troublesome implibs that you have listed above
(at least those that I have easy access to, but I have at least one
from each class of problems) and a few others. Except for MAPI.lib,
but my MS provided dumpbin.exe (VS 2005) says
  MAPI.lib : warning LNK4048: Invalid format file; ignored
for that one so I too think it is a pathological case.

Also, it will not fail for Vfw32.Lib, it will instead list the three
dlls it imports (AVICAP32.dll, AVIFIL32.dll, MSVFW32.dll).

Using MS tools (instead of file or objdump -f) to identify if a .lib
is an import lib or a static lib seems to be trickier. One thing
that appears to work is to look for an __IMPORT_DESCRIPTOR_* symbol,
but that can obviously be thwarted by a devious (or ignorant) user...

BTW, those symbols also identifies the imported dll (but that breaks
when that which is imported isn't named foo.dll). E.g.
  dumpbin -symbols $lib | grep '| __IMPORT_DESCRIPTOR_'

(output for Vfw32.Lib
  001 00000000 SECT2  notype       External     | __IMPORT_DESCRIPTOR_MSVFW32
  001 00000000 SECT2  notype       External     | __IMPORT_DESCRIPTOR_AVIFIL32
  001 00000000 SECT2  notype       External     | __IMPORT_DESCRIPTOR_AVICAP32
)

or, in gnu speak:
  nm $lib | grep 'I __IMPORT_DESCRIPTOR_'

(output for Vfw32.Lib
  00000000 I __IMPORT_DESCRIPTOR_MSVFW32
  00000000 I __IMPORT_DESCRIPTOR_AVIFIL32
  00000000 I __IMPORT_DESCRIPTOR_AVICAP32
)

> But...I also dislike for fixes to existing bugs, in existing platforms,
> to be held up by not-yet-in-master support for other compilers. So, can
> we get back to discussing the original patch, under the predicates of
> cygwin and mingw (not msvc) $hosts?

Hey, I'm not opposed to the patch, I didn't intend to make that impression
either, sorry if I did. I'm just trying to determine what needs to be
done in the MSVC branch to "keep up". Just poking and asking questions,
so thank you very much for your valuable input!

(However, as discussed earlier the MSVC branch is also using mingw as
 $host, but that is just a minor reminder which changes just about
 nothing in this context.)

Cheers,
Peter