[Savannah-hackers-public] Savannah Test Project

[Bob Proulx]

Savannah Hackers,

What is Legacy Code?  Hearing those words sends chills down most
developers spines.  Wikipedia says "source code inherited from someone
else and source code inherited from an older version of the software."
In his book Working Effectively with Legacy Code, Michael Feathers
defines Legacy Code as code without tests.  Because while it is always
easy to change code it is hard to do so without breaking things
because there is no way to know if something is actually broken.

One of the problems with Savannah is that it is quite a large project
that has evolved over literally decades.  Along the way it has
collected many features from all over.  Some large.  Some small.  It
is really difficult to create a complete list of what is the full
feature set of Savannah.  We keep learning about new features!

In order to help with this I have started a new Savannah
administration sub-project to run external tests against the site.
Since brute force usually triumphs over elegance and finesse I am
simply barging forward with a brute force set of external tests
trying to exercise every possible interface that we have available.

For example just testing the CVS pserver alone and nothing else
produces the following list of access combinations[1].  I picked the
hello project for this since it existed.  However that is really too
large for testing.  I plan to create a dedicated project for all of
these for testing purposes.  But this should give you an idea of the
amount of complexity that exists.  Each of those listed is used by
something and if any one of them stops working then it breaks a
feature for someone using that feature with at least one of the
projects hosted.  This cannot easily test that hooks are operating
properly and other such things.  But any test is better than nothing.

This testing project is still in the very early stages and I hesitate
to "announce" it publicly because it still needs work.  Regardless
here is the project.

  git clone ssh://git.savannah.gnu.org/srv/git/administration/savannah-tests.git
  cd savannah-tests
  make
  make check

Currently the biggest difficultly for other people is that it uses ssh
to connect to each of the many combinations of hostnames available.
Each of those (currently) must exist in the ssh known_hosts file or it
will ask for approval.  Obviously that needs to be polished.
Additionally USER must be set to an accessible Savannah username and
the ssh-agent must provide the ssh key for it.  If your Savannah
username is different then this is one way to set it.

  env USER=foo make check

By way of noting other hints it is easy to run individual tests.

  cd test
  make check TESTS=svn-web

Initially I put this together as a standalone external project test.
Often when the FSF admins need to do something fundamental we get
asked if we can check that Savannah is operating correctly.  This is a
first attempt to do so.  Longer term I am aiming at using Nagios or
Icinga2 or other for continuous checking.  As a project such as this
Jenkins could be used for this testing and it would simply look like
successful or failed builds.  Obviously it needs to be automated for
periodic operational checks.  Previously there has been a very few
basic Nagios tests that alert if a service fails but those simply
looked at one URL path as a very basic test and was not extensive
enough to cover what is really needed for migrations and for deep
health checking.

Bob

[1] List of just the CVS pserver access combinations:

    :pserver:address@hidden:/cvsroot/hello
    :pserver:address@hidden:/sources/hello
    :pserver:address@hidden:/srv/cvs/sources/hello
    :pserver:address@hidden:/srv/cvs/web/hello
    :pserver:address@hidden:/web/hello
    :pserver:address@hidden:/webcvs/hello
    :pserver:address@hidden:/cvsroot/hello
    :pserver:address@hidden:/sources/hello
    :pserver:address@hidden:/srv/cvs/sources/hello
    :pserver:address@hidden:/srv/cvs/web/hello
    :pserver:address@hidden:/web/hello
    :pserver:address@hidden:/webcvs/hello
    :pserver:address@hidden:/cvsroot/hello
    :pserver:address@hidden:/sources/hello
    :pserver:address@hidden:/srv/cvs/sources/hello
    :pserver:address@hidden:/srv/cvs/web/hello
    :pserver:address@hidden:/web/hello
    :pserver:address@hidden:/webcvs/hello
    :pserver:address@hidden:/cvsroot/hello
    :pserver:address@hidden:/sources/hello
    :pserver:address@hidden:/srv/cvs/sources/hello
    :pserver:address@hidden:/srv/cvs/web/hello
    :pserver:address@hidden:/web/hello
    :pserver:address@hidden:/webcvs/hello

SSH combinations:

    :ext:address@hidden:/cvsroot/hello
    :ext:address@hidden:/sources/hello
    :ext:address@hidden:/srv/cvs/sources/hello
    :ext:address@hidden:/srv/cvs/web/hello
    :ext:address@hidden:/web/hello
    :ext:address@hidden:/webcvs/hello
    :ext:address@hidden:/cvsroot/hello
    :ext:address@hidden:/sources/hello
    :ext:address@hidden:/srv/cvs/sources/hello
    :ext:address@hidden:/srv/cvs/web/hello
    :ext:address@hidden:/web/hello
    :ext:address@hidden:/webcvs/hello
    :ext:address@hidden:/cvsroot/hello
    :ext:address@hidden:/sources/hello
    :ext:address@hidden:/srv/cvs/sources/hello
    :ext:address@hidden:/srv/cvs/web/hello
    :ext:address@hidden:/web/hello
    :ext:address@hidden:/webcvs/hello
    :ext:address@hidden:/cvsroot/hello
    :ext:address@hidden:/sources/hello
    :ext:address@hidden:/srv/cvs/sources/hello
    :ext:address@hidden:/srv/cvs/web/hello
    :ext:address@hidden:/web/hello
    :ext:address@hidden:/webcvs/hello

Rsync combinations:

    rsync://cvs.savannah.gnu.org/sources/hello/
    rsync://cvs.savannah.nongnu.org/sources/hello/
    rsync://cvs.sv.gnu.org/sources/hello/
    rsync://cvs.sv.nongnu.org/sources/hello/

Web combinations using http:

    http://cvs.savannah.gnu.org/
    http://cvs.savannah.gnu.org/viewvc/
    http://cvs.savannah.gnu.org/viewcvs/
    http://cvs.savannah.nongnu.org/
    http://cvs.savannah.nongnu.org/viewvc/
    http://cvs.savannah.nongnu.org/viewcvs/
    http://cvs.sv.gnu.org/
    http://cvs.sv.gnu.org/viewvc/
    http://cvs.sv.gnu.org/viewcvs/
    http://cvs.sv.nongnu.org/
    http://cvs.sv.nongnu.org/viewvc/
    http://cvs.sv.nongnu.org/viewcvs/

Web combinations using https:

    https://cvs.savannah.gnu.org/
    https://cvs.savannah.gnu.org/viewvc/
    https://cvs.savannah.gnu.org/viewcvs/
    https://cvs.savannah.nongnu.org/
    https://cvs.savannah.nongnu.org/viewvc/
    https://cvs.savannah.nongnu.org/viewcvs/
    https://cvs.sv.gnu.org/
    https://cvs.sv.gnu.org/viewvc/
    https://cvs.sv.gnu.org/viewcvs/
    https://cvs.sv.nongnu.org/
    https://cvs.sv.nongnu.org/viewvc/
    https://cvs.sv.nongnu.org/viewcvs/

Now think about all of the combinations for each of the other vcs
programs operating at Savannah and you get the idea of the problem.