bug#7849: new instspc* test failures

[Stefano Lattarini]

Hi Peter, thanks for the super-quick feedback.

On 01/04/2012 03:17 PM, Peter Rosin wrote:
> Stefano Lattarini skrev 2012-01-04 13:24:
>> (Me rummaging in older bug reports ...)
>>
>> Reference:
>>   <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7849>
>>
>> On 01/16/2011 09:46 AM, Ralf Wildenhues wrote:
>>> There are a number of new failures since the splitup of instspc.test.
>>> Most seem to stem from increased coverage.  Rather than putting each
>>> failure in a different PR, I'm listing a number of them here; we can
>>> still split things out further later if that proves necessary.
>>>
>>> On MinGW, instspc-carriageret-* fail because the CR character is not
>>> treated correctly by the shell and/or other tools like sed.  I don't
>>> think this is fixable without fixing the shell.
>>>
>>> On Cygwin, instspc-dotdotdot-build.test fails because executing a COFF
>>> binary in a directory named '...' fails (causing the "compiler works"
>>> test to fail at configure time):
>>>
>>> $ mkdir ...
>>> $ cd ...
>>> $ cp /usr/bin/ls.exe .
>>> $ ./ls.exe
>>> bash: ./ls.exe: No such file or directory
>>>
>> Can anyone still reproduce such failures with the latest version of
>> `instspc.tap' from master?  If yes, can anyone suggest some workarounds
>> for failures that are due to bugs/limitations in Cygwin and MSYS rather
>> than to automake bugs?
> 
> Cygwin runs the test ok these days, but not MSYS/MinGW.
> MSYS still stumbles on carriageret and the only workaround I could
> think of is to check if the current test is carriageret and
> skip instead of fail if $test_string is empty in that case.
> But that's quite ugly,
>
Ah, but no more ugly than many other workarounds we have to avoid spurious
testsuite failure (if you want to see real horrors, take a look at, e.g.,
'uninstall-fail.test', or at 'tap-signal.tap' before commit d47115).

>    eval "test_string=\${instspc__$test_name}" \
>      && test x"$test_string" != x \
> -    || fatal_ "invalid test name: '$test_name'"
> +    || {
> +      test x"$test_name" != xcarriageret \
> +        && fatal_ "invalid test name: '$test_name'"
> +      skip_ "CR probably treated as NL: '$test_name'"
>
You need to skip *two* tests here, otherwise the driver will complain that the
number of tests run doesn't match the TAP plan.  Also, no need to relax the
sanity check on the `eval' (that should always work also on  MinGW).  I'd go
for something like this in the end (untested!):

   eval "test_string=\${instspc__$test_name}" \
     || fatal_ "invalid test name: '$test_name'"

   if test x"$test_string" != x; then
     if test x"$test_name" != xcarriageret; then
       fatal_ "invalid test name: '$test_name'"
     else
       # MinGW/MSYS version x.y still mishandles carriage returns; see
       # automake bug#7849.
       skip_ -r "carriage-return treated as null char" "$test_name in builddir"
       skip_ -r "carriage-return treated as null char" "$test_name in destddir"
       continue
     fi
   fi

> and doesn't really bring us much as the test falls over again on
> dosdrive aka 'a:'.
>
Well, it will bring us something once we've fixed also this error :-)

> On MSYS, "mkdir ./a:" creates
> the directory "a", w/o the trailing colon. Marvelous.
>
Indeed.

> So, next (ugly as hell) thing to do is to try to create a dir with
> only a colon if there is any colon in $test_string and skip if that
> fails.
>
Hmm...  what about this simpler sanity check instead?

  -mkdir "./$test_string" || {
  +# On MSYS, "mkdir ./a:" creates the directory "a", without the trailing
  +# colon.  Marvelous.  So, even if mkdir succeeds, we must still check that
  +# the create directory has the expected name.
  +mkdir "./$test_string" && test -d "./$test-string" || {
     skip_ -r "mkdir failed" "$test_name in builddir"
     skip_ -r "mkdir failed" "$test_name in destdir"
  ...

Would this work too?

> If you're not busy puking all over the place after reading the
> patch, you might be able to convince me to write a commit message
> for it...
>
Pretty please? ;-)

> The test no longer "fatals" on MSYS/MingW with these changes
> (instspc: exit 0).
>
Probably you already know, but I'd rather stress this anyway, just to be sure:
the TAP-based test scripts do *not* report failure/success through their final
exit status (any non-zero exit status will be flagged as an hard-error in fact);
the report of test results will be done with special "directives" emitted on
the stdout.  So the above result doesn't mean that `instspc.tap' is now passing,
but only that it is not encountering unexpected errors anymore.

To correctly execute a TAP-based script and get its results, you must either
run it through the automake-provided harness:

  $ make check TESTS=instspc.tap

or use a proper third-party TAP runner:

  $ prove --merge ./instspc.tap


Thanks,
  Stefano