Re: require_ulimit_v_(): tests skipped with enable-single-binary

[Pádraig Brady]

On 09/09/15 11:42, Pádraig Brady wrote:
> On 09/09/15 00:18, Bernhard Voelker wrote:
>> All tests using 'require_ulimit_v_' are skipped when configuring with
>> either --enable-single-binary=shebangs or ...=symlinks.
>>
>>
>> + require_ulimit_v_
>> + local ulimit_works=yes
>> + local vm
>> + case $(printenv LD_PRELOAD) in
>> ++ printenv LD_PRELOAD
>> + vm=10000
>> + ulimit -v 10000
>> + date
>> /tmp/coreutils-8.24.16-907f3/build2/src/coreutils: error while loading 
>> shared libraries: libcap.so.2: failed to map
>> segment from shared object: Cannot allocate memory
>> + ulimit_works=no
>> + ulimit -v 20
>> + date
>> ../init.cfg: line 148: 29355 Killed                  date
>> + test no = no
>> + skip_ 'this shell lacks ulimit support'
>>
>>
>> The reason is that the multicall 'coreutils' binary requires
>> much more memory due to shared libraries being loaded, and the
>> size of the 'date' binary (~290KiB) compared to the multicall
>> binary (~5MiB), of course.
>>
>> Adapting the above limit in the case the single-binary option is
>> active doesn't help much, as the actual tests would still use hard
>> coded values for 'ulimit -v' which are too low, of course.
>>
>> One solution could be that 'require_ulimit_v_' determines the
>> basic memory the actual test program (cp, dd, etc.) needs, and
>> that the test then uses that value plus a certain number as actual
>> memory limit.  Something along the following (completely untested!)
>> lines:
>>
>>   require_ulimit_v() {
>>     ...
>>     local v
>>     for v in $( seq 1000 1000 50000 ); do
>>       ( ulimit -v $v && "$@" ) && { echo $v; return 0; }
>>     done
> 
> 5000 looks like a base on Fedora 22 x86_64 here.
> Maybe this would be a better method:
> 
>     for v in $( seq 5000 5000 50000 ); do
>       if ( ulimit -v $v && "$@" ); then
>         prev_v=$v
>         for v in $( seq $(($prev_v-1000)) -1000 1000 ); do
>           ( ulimit -v $v && "$@" ) || { echo $prev_v; return 0; }
>           prev_v=$v
>         done
>       fi
>     done
>     return 1
> 
>>     ...
>>     skip_ "can not determine basic ulimit"
>>     ...
>>   }
>>
>> and in the tests:
>>
>>   base=$( require_ulimit_v_ dd if=/dev/zero of=/dev/null bs=1 count=1 ) \
>>     || skip_ "ulimit not working"
>>
>>   ( ulimit -v $(expr $base + 30000) && dd ... ) || fail=1
>>   ...
>>
>> WDYT?  Is this worth the trouble?
> 
> 
> I mentioned this as a TODO previously, so +1 from me.
> Having this adaptive is better anyway, as then all
> these tests are more adaptable to disparate and changing systems.
> 
> Given the number of tests this impacts I can't think
> of anything better than the above method.
> 
>   $ git grep -l ulimit tests/
>   tests/cp/link-heap.sh
>   tests/dd/no-allocate.sh
>   tests/misc/csplit-heap.sh
>   tests/misc/cut-huge-range.sh
>   tests/misc/head-c.sh
>   tests/misc/printf-surprise.sh
>   tests/misc/sort-continue.sh
>   tests/misc/sort-merge-fdlimit.sh
>   tests/misc/sort-spinlock-abuse.sh
>   tests/rm/many-dir-entries-vs-OOM.sh
>   tests/split/line-bytes.sh
>   tests/split/r-chunk.sh

Another gotcha with ulimit is that setting it too low
can disable any locale specific functionality,
because setlocale() will fail below 120M in testing here,
in which case we proceed in the "C" locale.

For example, testing the recent fix for the sort -M mem leak,
I was surprised that I couldn't trigger with:

  yes | (ulimit -v15000 && strace -e brk sort -c -M >/dev/null)

until I realized I needed to _increase_ the limit

  yes | (ulimit -v150000 && strace -e brk sort -c -M >/dev/null)

Now it's quite problematic that setlocale() failure can result
in different program behavior under memory pressure, but
also setlocale() doesn't distinguish between ENOMEM and ENOENT
(failure to lookup the LC_ALL entry), and given the likelihood
of the latter, we couldn't exit() upon selocale() failure I think.
Also warnings are not POSIX compliant without an associated exit(>0),
but the least we could do for the moment is to diagnose with --debug like:

diff --git a/src/sort.c b/src/sort.c
index c227f5b..dfd31f0 100644
--- a/src/sort.c
+++ b/src/sort.c
@@ -4833,10 +4833,11 @@ main (int argc, char **argv)
   char *files_from = NULL;
   struct Tokens tok;
   char const *outfile = NULL;
+  bool locale_ok;

   initialize_main (&argc, &argv);
   set_program_name (argv[0]);
-  setlocale (LC_ALL, "");
+  locale_ok = setlocale (LC_ALL, "");
   bindtextdomain (PACKAGE, LOCALEDIR);
   textdomain (PACKAGE);

@@ -5357,7 +5358,10 @@ main (int argc, char **argv)
         error (0, 0, _("using %s sorting rules"),
                quote (setlocale (LC_COLLATE, NULL)));
       else
-        error (0, 0, _("using simple byte comparison"));
+        {
+          error (0, 0, "%s%s", locale_ok ? "" : _("failed to set locale; "),
+                 _("using simple byte comparison"));
+        }
       key_warnings (&gkey, gkey_only);
     }


thanks,
Pádraig.

p.s. using valgrind for mem leak checking is less general
because it would depend on "lint" being defined to avoid all
"definitely lost" warnings.