[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: mdemo ltdl failure
|
From: |
Charles Wilson |
|
Subject: |
Re: mdemo ltdl failure |
|
Date: |
Sat, 17 Mar 2007 14:11:00 -0500 |
|
User-agent: |
Thunderbird 1.5.0.10 (Windows/20070221) |
Charles Wilson wrote:
Charles Wilson wrote:
I'll whip up a patch and post it to the newlib list.
So, I posted the following:
http://sourceware.org/ml/newlib/2007/msg00271.html
However, there's no telling how long it'll be before a new cygwin kernel
is released that includes the patch, and besides, libtool ought to work
on current and reasonably recent cygwin kernels. So, how should libtool
address this issue in the interim? Always use libltdl's argz stuff on
cygwin, until <further notice: version check on uname, etc etc>?
If so, how do we induce that
Here's my attempt:
(1) if argz is found on the system, perform a runtime test to see if the
system-provided argz facility actually works
(2) if so, everything is fine
(3) if not, then we need to use the libltdl-provided argz
(a) define a new symbol SYSTEM_ARGZ_IS_BROKEN
(b) set ARGZ_H and AC_LIBOBJ, just as if we didn't find
argz on the system
(c) use SYSTEM_ARGZ_IS_BROKEN in lt__glibc.h along with
HAVE_ARGZ_H to determine whether to
#define argz* lt__argz*
(4) if cross-compiling, be pessimistic: assume the target system's
argz facilities are broken, and use libltdl's argz -- see (3).
The tricky part is the runtime test itself -- detecting malloc
misbehavior is hard. I found a mechanism that reliably fails with
current cygwin/newlib, passes with a cygwin/newlib containing this fix
(http://sourceware.org/ml/newlib/2007/msg00271.html ), and passes on linux.
Even though I am pessimistic on cross-compiles, the test itself is
"optimistic": if it cannot set up malloc'ed memory in the manner it
expects, it returns success. The only hazard of a false failure, is if
the test program cannot compile.
So, here's how the test program works:
(1) allocate a big chunk of memory. Later, we will free this memory,
providing a nice target chunk for argz to realloc into.
We will try to "bracket" the memory allocated to argz initially. Most
mallocs, if a specific allocation needs to realloc larger, will avoid
copying by simply increasing the memory assigned to the current
allocation without moving it, if possible. We want to ensure that this
action is NOT possible, by allocating immovable memory immediately
before and immediately after the initial argz allocation. However,
arena-based mallocs can defeat this if our small "blocker" allocations
are in one arena, but the larger initial argz allocation is in a
different arena. We detect this situation and give up: return success
(e.g. go ahead and use system argz).
(2) So, allocate a "below" blocker chunk of memory
(3) use system argz facilities a bit -- this will cause argz allocations.
(4) Allocate an "above" blocker chunk of memory. Make sure the requested
size is larger than abs(argz - below). Otherwise, smart mallocs may
slot this blocker in between "below" and argz. However, check that
abs(argz - below) is not unreasonably large: arena mallocs may have
allocated "below" and argz from completely different arenaa, in totally
different locations. If abs(argz - below) is "big", assume that malloc
is "weird" in this way, and give up: return success (go ahead and use
system argz)
(5) make sure either
"below" < argz < "above", or
"above" < argz < "below"
otherwise, give up: system malloc is "wierd" -- arena based or
something. return success: go ahead and use system argz
(6) free the memory allocated in step (1), to provide a nice, tempting
chunk of memory for the next step to realloc into.
(7) Use argz_insert() to insert a really large chunk at the front of
*argz. In the failure mode I'm trying to detect, this will realloc argz
into the memory freed in step 6, just below the "below" blocker. The
buggy argz_insert will then copy from this uninitialized memory into the
OLD, now freed, argz memory. That's strike one. Strike two: it copies
(old)argz_len bytes, but offset by len(new_entry) -- which means it
overwrites past the end of the old *argz buffer by len(new_entry) bytes
-- and clobbers the book-keeping info and some of the contents of one of
our blockers. Strike three: freeing that blocker will coredump, since
its bookkeeping info is scrogged. (Actually, with a checking-malloc,
any one of the three strikes above will directly cause a coredump.)
Now, there are a bunch of ways that a broken argz could "get lucky" and
pass this test. That's okay: since this test is run on all platforms if
system argz is found, I wanted to be lenient: few to zero false
failures. Because we've introduced a new symbol (and I use AC_CACHE),
the following remedies are available:
(1) Hopefully this never happens, but a false failure report could be
overridden by setting $lt_cv_sys_argz_works=yes, and the runtime test
will be skipped.
(2) It's possible that on some (broken) cygwin system, the malloc
might not behave as expected and the test program return success. This
can be overridden by setting $lt_cv_sys_argz_works=no.
(3) Once cygwin is fixed, the libtool distributor (me) might want to
ensure that the distributed libltdl is usable on systems with an older
(unfixed) cygwin kernel. See #2.
(4) On cross-compile, argz.m4 is pessimistic, and assumes the target
system argz is broken. You can force the use of the target system argz
-- see (1). This is a little odd: even though the autoconf
documentation recommends pessimism for AC_TRY_RUN's cross-compile
clause, perhaps we should be optimistic instead?
=======
One other change: you can now force the use of libltdl's argz on any
system, which was not possible before when HAVE_ARGZ_H was true. BUT,
doing this with HAVE_ARGZ_H true carries some risk: we've already
decided whether to #define error_t ourselves, and/or set
__error_t_defined, based on what we detected AFTER #including the
system's argz.h -- but by forcing the use of libltdl's argz, we won't
use the system's argz.h, so at compile-time we might not "see" what the
error_t test "saw".
Therefore, libltdl's argz_.h needs to include most of the common places
where error_t may have been "picked up" by the system argz.h. On newlib
systems, this is <errno.h>. On glibc systems, this is also <errno.h>,
but you must #define __need_error_t first (which glibc's argz.h does).
So, I've modified libltdl's argz_.h to do that, too.
========
I'm still using the -export-symbols-regex ".*" fix for libmlib in
tests/mdemo -- it's also included in this patch.
TESTING:
I've tested this under the following conditions:
(1) broken cygwin kernel
--detects failure, builds successfully using libltdl's argz
--resultant libraries also work after dropping in a
fixed cygwin kernel.
(2) fixed cygwin kernel
--detects success, builds successfully using system argz
--resultant libraries coredump if you drop in a broken cygwin
kernel after the fact. This is expected: broken cygwin is
"broken" precisely because its argz facility coredumps on
argz_insert().
(3) fixed cygwin kernel, but with $lt_cv_sys_argz_works=no.
--does not even try to run the test program, and successfully
builds using libltdl's argz
--resultant libraries works fine even after dropping in a
broken cygwin kernel.
(4) linux (whose system argz is OK)
--detects success, builds using system argz, works.
(5) linux, but with $lt_cv_sys_argz_works=no.
--doesn't run the test, and builds using libltdl argz. works.
(6) mingw, which doesn't have any system argz facility at all
--the test is not run
--builds successfully with libltdl's argz
Full test suite results:
cygwin, broken kernel (case 1, above):
All 115 tests passed (old test suite)
[libtool 2.1a] testsuite: 14 16 33 34 35 49 failed
16, 49: known failures on this platform
14: java -- I dunno if gcj even works on cygwin
33,34,35: new regressions in CVS between 20070205 and 20070316
but I get these failures both with and without the
attached patch
mingw: (case 6, above)
All 115 tests passed (old test suite)
[libtool 2.1a] testsuite: 14 16 19 23 35 39 42 49 failed
16, 49: known failures on this platform
14: java -- I dunno if gcj even works on mingw
23,35| anything that uses DESTDIR is broken on msys
39,42| because C:/DESTDIR + C:/PREFIX == C:/DESTDIRC:/PREFIX
and that's just not right.
19: static.at -- I think this is pre-existing.
CHANGELOG:
2007-03-17 Charles Wilson <address@hidden>
* libltdl/argz_.h: ensure error_t definition is obtained
in same mechanism system argz.h would have.
* libltdl/libltdl/lt__glibc.h: also detect if
SYSTEM_ARGZ_IS_BROKEN when determining whether to re#def
argz* functions.
* libltdl/m4/argz.m4: add new runtime test to check if
system argz facilities actually work. Be lenient within
test, but pessimistic on cross-compile.
* tests/mdemo/Makefile.am: -export-symbols-regex fix for
libmlib.
--
Chuck
Index: libltdl/argz_.h
===================================================================
RCS file: /cvsroot/libtool/libtool/libltdl/argz_.h,v
retrieving revision 1.6
diff -u -r1.6 argz_.h
--- libltdl/argz_.h 4 Sep 2006 17:23:30 -0000 1.6
+++ libltdl/argz_.h 17 Mar 2007 06:09:46 -0000
@@ -31,6 +31,8 @@
#define LT__ARGZ_H 1
#include <stdlib.h>
+#define __need_error_t
+#include <errno.h>
#include <sys/types.h>
#if defined(LTDL)
Index: libltdl/libltdl/lt__glibc.h
===================================================================
RCS file: /cvsroot/libtool/libtool/libltdl/libltdl/lt__glibc.h,v
retrieving revision 1.7
diff -u -r1.7 lt__glibc.h
--- libltdl/libltdl/lt__glibc.h 26 Oct 2006 20:39:04 -0000 1.7
+++ libltdl/libltdl/lt__glibc.h 17 Mar 2007 06:09:49 -0000
@@ -36,7 +36,7 @@
# include <config.h>
#endif
-#if !defined(HAVE_ARGZ_H)
+#if !defined(HAVE_ARGZ_H) || defined(SYSTEM_ARGZ_IS_BROKEN)
/* Redefine any glibc symbols we reimplement to import the
implementations into our lt__ namespace so we don't ever
clash with the system library if our clients use argz_*
Index: libltdl/m4/argz.m4
===================================================================
RCS file: /cvsroot/libtool/libtool/libltdl/m4/argz.m4,v
retrieving revision 1.3
diff -u -r1.3 argz.m4
--- libltdl/m4/argz.m4 25 Mar 2006 11:05:02 -0000 1.3
+++ libltdl/m4/argz.m4 17 Mar 2007 06:09:50 -0000
@@ -27,6 +27,142 @@
ARGZ_H=
AC_CHECK_FUNCS([argz_append argz_create_sep argz_insert argz_next \
argz_stringify], [], [ARGZ_H=argz.h; AC_LIBOBJ([argz])])
+
+dnl if success, do a runtime check to ensure it actually works
+if test -z "$ARGZ_H"; then
+ AC_CACHE_CHECK(
+ [if argz actually works],
+ [lt_cv_sys_argz_works],[dnl
+AC_TRY_RUN([
+#include <stddef.h>
+#include <stdlib.h>
+#include <string.h>
+#if defined(HAVE_ARGZ_H)
+# include <argz.h>
+#endif
+int main()
+{
+ error_t e;
+ const char* in = "a long string so that we will eventually realloc;"
+ "and another entry that is fairly long";
+ const char* add = "add this to the end of the argz vector";
+ const char* insert = "insert at the beginning. This should force a
realloc";
+ char* argz = 0;
+ size_t len = 0;
+ size_t cnt = 0;
+ char* some_freed_space = 0;
+ char* blocker_low = 0;
+ char* blocker_high = 0;
+ int rc = 0;
+ ptrdiff_t low_sign = 0;
+ ptrdiff_t high_sign = 0;
+
+ some_freed_space = (char*)malloc(1000*sizeof(char));
+ strcpy(some_freed_space, "SOME FREED SPACE");
+
+ blocker_low = (char*)malloc(20*sizeof(char));
+ strcpy(blocker_low, "DUMMY BLOCKER LOW");
+
+ e = argz_create_sep(in, ';', &argz, &len);
+ if (e)
+ {
+ free(blocker_low);
+ free(some_freed_space);
+ return 1;
+ }
+
+ low_sign = (blocker_low - argz > 0 ? 1 : -1);
+
+ cnt = argz_count(argz, len);
+ if (cnt != 2)
+ {
+ free(argz);
+ free(blocker_low);
+ free(some_freed_space);
+ return 1;
+ }
+ e = argz_add(&argz, &len, add);
+ if (e)
+ {
+ free(argz);
+ free(blocker_low);
+ free(some_freed_space);
+ return 1;
+ }
+ cnt = argz_count(argz, len);
+ if (cnt != 3)
+ {
+ free(argz);
+ free(blocker_low);
+ free(some_freed_space);
+ return 1;
+ }
+
+ {
+ ptrdiff_t fillup = argz - blocker_low;
+ if (fillup < 0) fillup = -fillup;
+ if (fillup > 10000)
+ {
+ /* give up and assume argz works; we don't understand
+ malloc's memory strategy */
+ free(some_freed_space);
+ free(blocker_low);
+ free(argz);
+ return 0;
+ }
+ blocker_high = (char*)malloc(2*fillup*sizeof(char));
+ strncpy(blocker_high, "DUMMY BLOCKER HIGH", 2*fillup);
+ }
+
+ high_sign = (blocker_high - argz > 0 ? 1 : -1);
+ /* if high_sign and low_sign are the same, the our blockers have
+ not boxed in argv. Give up, and assume argz works */
+ if ( high_sign * low_sign > 0 )
+ {
+ free(some_freed_space);
+ free(blocker_high);
+ free(blocker_low);
+ free(argz);
+ return 0;
+ }
+
+ /* now free the freed space in hopes that realloc will use it */
+ free(some_freed_space);
+
+ e = argz_insert(&argz, &len, argz, insert);
+ if (e)
+ {
+ free(argz);
+ free(blocker_low);
+ free(blocker_high);
+ return 1;
+ }
+ cnt = argz_count(argz, len);
+ if (cnt != 4)
+ {
+ free(argz);
+ free(blocker_low);
+ free(blocker_high);
+ return 1;
+ }
+
+ free(argz);
+ free(blocker_low);
+ free(blocker_high);
+ return 0;
+}],
+[lt_cv_sys_argz_works=yes],
+[lt_cv_sys_argz_works=no],
+[lt_cv_sys_argz_works=unknown-assuming-no])
+])
+AS_IF([test $lt_cv_sys_argz_works != yes],
+ [AC_DEFINE([SYSTEM_ARGZ_IS_BROKEN], 1,
+ [This value is set to 1 to indicate that the system argz
facility does not work])
+ ARGZ_H=argz.h
+ AC_LIBOBJ([argz])],
+[])
+fi
+
AC_SUBST([ARGZ_H])
])
Index: tests/mdemo/Makefile.am
===================================================================
RCS file: /cvsroot/libtool/libtool/tests/mdemo/Makefile.am,v
retrieving revision 1.9
diff -u -r1.9 Makefile.am
--- tests/mdemo/Makefile.am 26 Feb 2007 07:44:25 -0000 1.9
+++ tests/mdemo/Makefile.am 17 Mar 2007 06:09:50 -0000
@@ -37,7 +37,7 @@
libmlib_la_SOURCES = mlib.c
libmlib_la_LIBADD = @LIBLTDL@ "-dlopen" foo1.la "-dlopen" libfoo2.la
-libmlib_la_LDFLAGS = -no-undefined
+libmlib_la_LDFLAGS = -no-undefined -export-symbols-regex ".*"
libmlib_la_DEPENDENCIES = @LIBLTDL@ libsub.la foo1.la libfoo2.la
noinst_HEADERS = foo.h