a package for low-level model profiling on Redhat 7.1

[Marcus G. Daniels]

In the Java models I've been running lately, I've been faced with
significant performance problems.  In particular, I've been wanting to
have a way to see what's happening in the computer in the `standard
Unix way' of setting up a clock and sampling the program counter.
Sure, Java has some simple profiling tools that show you what's
happening `above' the JVM, but nothing that approximates the nice
bits-between-your-toes feel of `gprof'.

Unfortunately, the Java world of dynamic loading of bytecodes and native
libraries is at odds with the world of nail-everything-down-and-watch world
of Unix profiling.

On Linux-based systems that use the GNU C library, there is some dynamic
linker magic that will help to startup the instrumentation of libraries
at runtime, but it is tricky to shoehorn it into the JVM environment.
For systems like Solaris, it is pretty much a hopeless endeavour.  In any
case, on every system, the solution is different..  [argh!]

And even _if_ you can get native executable (e.g. compiling with the
GNU Java compiler, gcj), in practice the problem is the same: shared
libraries and profiling are hard or impossible to do together.

As you might expect, profiling of Java code and Java infrastructure is
problem other researchers have had.  And indeed there exists free
source code to do tackle the problem.

For example, Kaffe can be configured to do fully static linkage of Java
native-code libraries _and_ external JNI native code libraries using
emulation code that makes the JVM believe is using dynamic linking
even though it is just an interface that looks like dynamic linking.
Some researchers at the University of Utah have enhanced this further to
instrument just-in-time compiled code (JIT) with profiling notes.

Happily, this approach works, and I'm using it now!

Since it is very time consuming and delicate to configure all the
packages that are needed to instrument Swarm models in this way, I've
uploaded my Redhat 7.1 profiling subtree, here:

 ftp://ftp.swarm.org/pub/swarm/binaries/redhat/i386/profiling/prof.tar.bz2

Usage:

1. Untar it in /packages to get /packages/prof.

2. Put /packages/prof/bin in the PATH and set SWARMHOME to /packages/prof

3. Run something:
  
   javaswarm -Xxprof StartHeatbugs -b

   Note: this package doesn't include X11 instrumented for profiling.
   If you are trying to make a GUI go fast, this won't provide a complete
   picture, although it will provide some, considering that all of Swarm,
   and Tk are instrumented. 

   For batch runs, the data is complete.

4. Profile it:

   kaffexprof kaffe-jit-symbols xgmon.out > prof.txt

5. Finally, read and think-about what's in prof.txt!
   (See the man page on `gprof' for info about the format.)

Below are my build notes.  You shouldn't have to mess with this stuff,
with the exception of the depend.m references.  I don't think it is
possible without significant extra work to port Kaffe's xprof capability
to systems besides FreeBSD or Linux.  I've used Redhat 7.1 and FreeBSD 4.0.

Building a static binary of Kaffe including Swarm (and dependencies):

*  Compile as many packages as posssible with profiling.  To get a pure
   static executable on Linux, I built the following with -pg and 
   various hacks to get static _only_ libraries.
   Not all of these packages will build as static libraries without 
   some further coaxing.  Watch carefully..  

   The packages are:

   BLT2.4u, jpeg v6b, gmp 3.3, libpng 1.0.11, zlib 1.1.3, libungif 3.0,
   tcl8.3.3, tk8.3.3, hdf5 1.2.2, and Swarm.  I didn't build X11 because
   it is a large, time-consuming build, and because it comes with
   static libraries.  This means that X11 will be invisible in the
   profile (and can be significant).  My concern is 
   profiling the performance of batch runs, so I don't care.

*  Because of the dynamic linking emulation works in Kaffe, it is necessary 
   to use libkaffeswarmstubs.la as a convenience library -- remove the -rpath
   in the libkaffeswarmstubs.la link in java/c/Makefile. 

   Also remove the install target for libkaffeswarmstubs.la.  It will all
   be merged into libkaffeswarm.la, which the Kaffe link will consume
   noting the Swarm symbols.

*  Build a version of Kaffe equipped with `xprofiling'.
   First you'll need a basic Kaffe install in the same install tree for
   building Swarm.  Once you've got Swarm, then you want to reconfigure
   Kaffe for static profiling, and that requires pulling Swarm and the other
   libraries into Kaffe itself.  I use a configure script like so:

#!/bin/sh
SRCDIR=$S/kaffe
INST=$P/prof
JAVA_LIBS=$INST/lib/swarm/libkaffeswarm.la CC=/usr/bin/gcc $SRCDIR/configure 
--prefix=$INST --srcdir=$SRCDIR --with-staticvm --with-staticlib 
--enable-xprofiling --with-profiling --with-includes=$INST/include 
--with-libraries=$INST/lib && make

After configuration, make these changes:

  A. LDADD additions in kaffe/kaffe/Makefile
  
    1. Add libswarm.la and libavcall.la to Kaffe link LDADD

    2. For an -all-static build, also add $(AWT_LIBS) -lgmp

    3. Use depend.m to pull in requisite Objective C classes, compile with
       "gcc -c depend.m" in kaffe/kaffe subdirectory

       Add depend.o to the LDADD list.

      This file pulls in references to Swarm classes that you are using.
      Obviously, many things are missing..   It should be possible to 
      relink the provided Kaffe executable with more references without
      doing this full build procedure.

#include <objc/Object.h>
@class GUISwarm;
@class EmptyProbeMap;
@class Diffuse2d;
@class Grid2d;
@class Colormap;
@class ZoomRaster;
@class Value2dDisplay;
@class Object2dDisplay;
@class EZGraph;
@class NormalDist;
@class EZBin;
@class RandomBitDist;
@class NSelect;
@class ProbeDisplayManager;
test ()
{
[GUISwarm self];
[EmptyProbeMap self];
[Diffuse2d self];
[Grid2d self];
[Colormap self];
[ZoomRaster self];
[Value2dDisplay self];
[Object2dDisplay self];
[EZGraph self];
[NormalDist self];
[EZBin self];
[RandomBitDist self];
[NSelect self];
[ProbeDisplayManager self];
}
 
 B.  Fix libtool in Kaffe to allow $ in names:
     Add \$ to second [_A-Z] pair in global_symbol_pipe 
     Do this immediately after `configure', but before `make'

 C.  On FreeBSD, I added -L$(P)/lib to export_dynamic_flag_spec  
     because it is early in the link command and will
     will force override of /usr/local  -- this gets the static zlib and png
     Other libraries are still dynamic, e.g. tcl, tk, gmp
     Alternately, take the approach of doing an -all-static build.

 D.  Linux needs an -all-static build in order to resolve certain profiling  
     symbols.
  
     (in kaffe/kaffe/Makefile add -all-static to KAFFE_LDFLAGS)

                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.