Re: Contradictory results from u_test and sign_test

[Rik]

> 
> Message: 2
> Date: Fri, 15 Jul 2011 15:52:14 +0200
> From: Alexander Klein <address@hidden>
> To: address@hidden
> Subject: Contradictory results from u_test und sign_test
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=iso-8859-1
> 
> Hi,
> 
> a while ago I noticed that u_test and sign_test produce quite contradictory 
> results:
> 
> octave-3.2.3:1> X=[1 2 3]
> X =
> 
>   1   2   3
> 
> octave-3.2.3:2> Y=[4 5 6]
> Y =
> 
>   4   5   6
> 
> octave-3.2.3:3> sign_test (X,Y,"<")
> warning: binomial_cdf is obsolete and will be removed from a future version 
> of Octave; please use binocdf instead
>  pval: 0.125
> ans =  0.12500
> octave-3.2.3:4> u_test (X,Y,"<")
>  pval: 0.975233
> ans =  0.97523
> 
> I contacted the original author, Kurt Hornik, but since he wrote the 
> functions quite some time ago he couldn't remember the details, yet agreed 
> that at least one result is questionable.
> 
Alex,

I think there is an issue here, but it may not be as big as it first
appears.  First, sign_test doesn't even run under the development version
of Octave because binomial_cdf has been removed.  In general, all of the
scripts in statistics/tests have not seen any attention in years and it
shows.  I have updated the scripts in statistics/base and am almost done
with those in statistics/distributions, but am running out of energy.  If
you could do anything to help these functions it would be appreciated.

As for your particular question.  I think sign_test is returning not the p
value but (1-p).  Secondly, if you read the documentation for each test,
sign_test assumes the test statistic follows a binomial distribution while
u_test assumes the test statistic follows a standard normal distribution.
By the central limit theorem, the binomial distribution can be approximated
by the normal distribution if N is large enough.  With that in mind, I
tried using 15 data points instead of 3.  See the following.

x = 1:15;
y = 16:30;
pval_sign = sign_test (x,y,'<')
  pval_sign =  3.05175781250000e-05
1 - pval_sign
  ans =  0.999969482421875
pval_utest = u_test (x,y,'<')
  pval_utest =  0.999998466511117

The agreement seems not so bad in this case.

Cheers,
Rik