Re: require_ulimit_v_(): tests skipped with enable-single-binary

[Pádraig Brady]

On 18/09/15 12:47, Pádraig Brady wrote:
> On 09/09/15 11:42, Pádraig Brady wrote:
>> On 09/09/15 00:18, Bernhard Voelker wrote:
>>> All tests using 'require_ulimit_v_' are skipped when configuring with
>>> either --enable-single-binary=shebangs or ...=symlinks.
>>>
>>>
>>> + require_ulimit_v_
>>> + local ulimit_works=yes
>>> + local vm
>>> + case $(printenv LD_PRELOAD) in
>>> ++ printenv LD_PRELOAD
>>> + vm=10000
>>> + ulimit -v 10000
>>> + date
>>> /tmp/coreutils-8.24.16-907f3/build2/src/coreutils: error while loading 
>>> shared libraries: libcap.so.2: failed to map
>>> segment from shared object: Cannot allocate memory
>>> + ulimit_works=no
>>> + ulimit -v 20
>>> + date
>>> ../init.cfg: line 148: 29355 Killed                  date
>>> + test no = no
>>> + skip_ 'this shell lacks ulimit support'
>>>
>>>
>>> The reason is that the multicall 'coreutils' binary requires
>>> much more memory due to shared libraries being loaded, and the
>>> size of the 'date' binary (~290KiB) compared to the multicall
>>> binary (~5MiB), of course.
>>>
>>> Adapting the above limit in the case the single-binary option is
>>> active doesn't help much, as the actual tests would still use hard
>>> coded values for 'ulimit -v' which are too low, of course.
>>>
>>> One solution could be that 'require_ulimit_v_' determines the
>>> basic memory the actual test program (cp, dd, etc.) needs, and
>>> that the test then uses that value plus a certain number as actual
>>> memory limit.  Something along the following (completely untested!)
>>> lines:
>>>
>>>   require_ulimit_v() {
>>>     ...
>>>     local v
>>>     for v in $( seq 1000 1000 50000 ); do
>>>       ( ulimit -v $v && "$@" ) && { echo $v; return 0; }
>>>     done
>>
>> 5000 looks like a base on Fedora 22 x86_64 here.
>> Maybe this would be a better method:
>>
>>     for v in $( seq 5000 5000 50000 ); do
>>       if ( ulimit -v $v && "$@" ); then
>>         prev_v=$v
>>         for v in $( seq $(($prev_v-1000)) -1000 1000 ); do
>>           ( ulimit -v $v && "$@" ) || { echo $prev_v; return 0; }
>>           prev_v=$v
>>         done
>>       fi
>>     done
>>     return 1
>>
>>>     ...
>>>     skip_ "can not determine basic ulimit"
>>>     ...
>>>   }
>>>
>>> and in the tests:
>>>
>>>   base=$( require_ulimit_v_ dd if=/dev/zero of=/dev/null bs=1 count=1 ) \
>>>     || skip_ "ulimit not working"
>>>
>>>   ( ulimit -v $(expr $base + 30000) && dd ... ) || fail=1
>>>   ...
>>>
>>> WDYT?  Is this worth the trouble?
>>
>>
>> I mentioned this as a TODO previously, so +1 from me.
>> Having this adaptive is better anyway, as then all
>> these tests are more adaptable to disparate and changing systems.
>>
>> Given the number of tests this impacts I can't think
>> of anything better than the above method.
>>
>>   $ git grep -l ulimit tests/
>>   tests/cp/link-heap.sh
>>   tests/dd/no-allocate.sh
>>   tests/misc/csplit-heap.sh
>>   tests/misc/cut-huge-range.sh
>>   tests/misc/head-c.sh
>>   tests/misc/printf-surprise.sh
>>   tests/misc/sort-continue.sh
>>   tests/misc/sort-merge-fdlimit.sh
>>   tests/misc/sort-spinlock-abuse.sh
>>   tests/rm/many-dir-entries-vs-OOM.sh
>>   tests/split/line-bytes.sh
>>   tests/split/r-chunk.sh
> 
> Another gotcha with ulimit is that setting it too low
> can disable any locale specific functionality,
> because setlocale() will fail below 120M in testing here,
> in which case we proceed in the "C" locale.
> 
> For example, testing the recent fix for the sort -M mem leak,
> I was surprised that I couldn't trigger with:
> 
>   yes | (ulimit -v15000 && strace -e brk sort -c -M >/dev/null)
> 
> until I realized I needed to _increase_ the limit
> 
>   yes | (ulimit -v150000 && strace -e brk sort -c -M >/dev/null)
> 
> Now it's quite problematic that setlocale() failure can result
> in different program behavior under memory pressure, but
> also setlocale() doesn't distinguish between ENOMEM and ENOENT
> (failure to lookup the LC_ALL entry), and given the likelihood
> of the latter, we couldn't exit() upon selocale() failure I think.
> Also warnings are not POSIX compliant without an associated exit(>0),
> but the least we could do for the moment is to diagnose with --debug like:
> 
> diff --git a/src/sort.c b/src/sort.c
> index c227f5b..dfd31f0 100644
> --- a/src/sort.c
> +++ b/src/sort.c
> @@ -4833,10 +4833,11 @@ main (int argc, char **argv)
>    char *files_from = NULL;
>    struct Tokens tok;
>    char const *outfile = NULL;
> +  bool locale_ok;
> 
>    initialize_main (&argc, &argv);
>    set_program_name (argv[0]);
> -  setlocale (LC_ALL, "");
> +  locale_ok = setlocale (LC_ALL, "");
>    bindtextdomain (PACKAGE, LOCALEDIR);
>    textdomain (PACKAGE);
> 
> @@ -5357,7 +5358,10 @@ main (int argc, char **argv)
>          error (0, 0, _("using %s sorting rules"),
>                 quote (setlocale (LC_COLLATE, NULL)));
>        else
> -        error (0, 0, _("using simple byte comparison"));
> +        {
> +          error (0, 0, "%s%s", locale_ok ? "" : _("failed to set locale; "),
> +                 _("using simple byte comparison"));
> +        }
>        key_warnings (&gkey, gkey_only);
>      }
> 
> 
> thanks,
> Pádraig.
> 
> p.s. using valgrind for mem leak checking is less general
> because it would depend on "lint" being defined to avoid all
> "definitely lost" warnings.

BTW the 100MB virtual mem size increase from setlocale()
is caused by mmaping the pre-processed locale archive
at /usr/lib/locale/locale-archive.  That's CoW'd and so
doesn't use any extra RAM per process, but still
impacts ulimit -v.

There is some work to reduce that system wide recently with:
https://fedoraproject.org/wiki/Changes/Glibc_locale_subpackaging

Though we may have more control over a test environment
by creating a much smaller subset (with en_US, C.UTF-8, fr_FR.UTF-8, ...)
using the build-locale-archive command on a separate archive,
and then getting setlocale() to reference it using the LOCPATH env variable.

thanks,
Pádraig.