Re: [Monotone-devel] Patch to add Composite Benchmarks and some simple scalable add/commit tests

[Nathaniel Smith]

On Mon, Jul 17, 2006 at 11:55:46PM -0700, Eric Anderson wrote:
> The idea behind the composite benchmark is that one can say:
> benchmark(setup-empty-workdir, setup-create-random-files, measure-add, 
>           measure-commit, measure-pull)
> 
> and the benchmark suite will first setup an empty working directory,
> create a bunch of random files in that working directory and then
> measure (using the instrumenter) the time to add, commit and pull the
> resulting repository.  This allows one to construct a series of
> operations and test them all in sequence.  The idea, although not
> implemented is that one could change measure- to setup- and the work
> would happen during setup not during measurement so that an expensive
> instrumenter would only be run over the relevant parts.
> 
> The precise CLI syntax for the above would look like:
>  -b simple-add='CompositeBenchmark(SetupEmptyWorkdir(), 
> SetupWorkdirAddFileSource(PartRandomFileSource(1,1024,0.5)), MeasureAdd(), 
> MeasureCommit(), MeasureWorkingPull())'

I still haven't had a chance to review the patch properly, sorry.
Having thought about it a bit, though, I'm feeling uncomfortable about
this general approach.  In particular, it is reminding me of the ad
hoc shell scripts that I was hoping to escape with this framework,
because it defines an oddly limited language for specifying procedural
computation (why am I writing a list of pre-defined steps, when I
could just be writing python code?), and it doesn't make explicit the
actual requirements (e.g., the Add benchmark needs a workspace, but
nothing here makes that obvious).  Finally, as a general note, I am
wary of tests that set up the environment for the next test, because
they are harder to reason about, harder to run individually or tweak
individually, and so on.

The general idea of composing mix-and-match parts is very much where I
want to go, though!  The vague plan I've been working towards is
currently illustrated by the "Repo" interface, as implemented by
"EmptyRepo" and "ExistingRepo" and, hopefully soon,
"SimpleRandomRepo".  These all implement particular interface, which
is documented, and then the Pull benchmark requires a Repo.  So you
can say "Pull(EmptyRepo())" or "Pull(SimpleRandomRepo(treesize=10**5))"
or whatever.  I was planning to create a similar sort of interface for
benchmarks that require a workspace, or a workspace + a way to
generate edits in the workspace, etc.

The overall result is very similar to your example above, but I think
better factored -- basically the difference is instead of collapsing
things that set up environments and things that actually run tests
into one concept, trying to actually identify the different sorts of
environmental requirements different tests have, and implementing
those seperately.

Of course, I haven't actually _done_ most of this yet, so who knows
if it will work.  But does that general Grand Plan make sense?

As for the specific case of porting existing tests, I think the
solution might be to just to create a way for tests to give names to
particular calls to the vcs object...?

-- Nathaniel

-- 
- Don't let your informants burn anything.
- Don't grow old.
- Be good grad students.
  -- advice of Murray B. Emeneau on the occasion of his 100th birthday