[cp-patches] Serializable.readResolve() and protected accessibility

[Andrew Haley]

This was promted by the failure of the entire JOnAS testsuite!

In the end, it turns out that readResolve() was not called by
ObjectStreamClass because we incorrectly check the accessibility of
the readResolve() method.  The readResolve() method in question was
protected, but in that case we carefully check to make sure that it is
in the same package as the caller.  But we don't need to do that:
protected accessibility includes all subclasses, not just those in the
same package.

A simple tescase makes the point:

t1.java:
public class t1
{
  public static void main (String[] argv)
    throws Exception
  {
    PipedOutputStream out = new PipedOutputStream();
    PipedInputStream in = new PipedInputStream(out);
    ObjectOutput o = new ObjectOutputStream(out);
    ObjectInput i = new ObjectInputStream(in);
    o.writeObject(new Bash.Thing());
    Object obj = i.readObject();
    System.out.println(obj);
  }
}

Booz/Barf.java:
package Booz;

import java.io.*;
import java.util.*;

public class Barf extends Date
{
  protected Object readResolve()
    throws ObjectStreamException
  {
    System.err.println("readResolve()");
    return new Object();
  }
}

Bash/Thing.java:
package Bash;

import java.io.*;
import java.util.*;

public class Thing extends Booz.Barf
{
}

So, with Sun readResolve() is called:

 $ java t1
readResolve()
address@hidden

And with Classpath it isn't:

 $ /usr/bin/gij t1
Fri Sep 16 11:51:48 GMT+00:00 2005

I took the opportunity to rewrite the logic used to check
accessibility: it was very hard to understand because it was formed as
"if this, DON'T do that".  It is always much harder to understand
negative guards than positive ones, which is, I suspect, why the code
was incorrect.

With this code, I can get a good set of JOnAS test reults.
http://people.redhat.com/~aph/jeremie-Linux-HSQL1-1.4.2-$%7Bwebcontainer.name%7D.html
Executive summary: Success rate 92.92%

Andrew.


2005-09-16  Andrew Haley  <address@hidden>

        * java/io/ObjectStreamClass.java (findAccessibleMethod): Allow
        protected readResolve().  Rewrite accessibility check.

--- java/io/ObjectStreamClass.java      2005-08-11 14:34:06.000000000 +0100
+++ /home/aph/gcc/gcc/libjava/java/io/ObjectStreamClass.java    2005-09-16 
13:13:31.000000000 +0100
@@ -514,14 +514,15 @@
          {
            Method res = c.getDeclaredMethod(name, noArgs);
            int mods = res.getModifiers();
-
-           if (c != from
-               && (Modifier.isPrivate(mods)
-                   || ! Modifier.isPublic(mods) && ! inSamePackage(c, from)))
-             continue;
-
-           AccessController.doPrivileged(new SetAccessibleAction(res));
-           return res;
+           
+           if (c == from  
+               || Modifier.isProtected(mods)
+               || Modifier.isPublic(mods)
+               || (! Modifier.isPrivate(mods) && inSamePackage(c, from)))
+             {
+               AccessController.doPrivileged(new SetAccessibleAction(res));
+               return res;
+             }
          }
        catch (NoSuchMethodException e)
          {