Re: Regression Testing

[Håkon Løvdal]

On 9/2/05, Sven Luther <address@hidden> wrote:
> On Fri, Sep 02, 2005 at 02:55:03PM +0200, address@hidden wrote:
> >
> > Dear fellow developers,
> >
> > I'd really like to get the regression testing thing sorted out.
> > Without it we always have to test basic functionality before
> > each release manually. This is error-prone and time-consuming, both of
> > which I am very fond of.
> > Let's discuss which system to use and then get it up and running.
> 
> you wnt to regression test the library, or the parted frontend too ?
> 
> We could have a set of special library test programs, which we would run on a
> set of disk images of each different partition table formats. Not sure how we
> can handle huge disk images though, so i think we would not be able to test
> the kind of breakage involved with big disks, unless we use files-with-holes
> or whatever those are called.

Swiss-cheese-like files?

On the more serious side, let me share my thoughts on regression testing.

A few years ago I wrote the program printpar which can print the partition
table in various ways. I have not done anything with printpar since then
but one of the things I really wanted (and still want) was regression
testing, so I spent some time thinking about that even though I did not
implement it.

I advocate creating a specific test file format containing (only)
the relevant sectors from the disk along with some corresponding
meta-data. The meta-data would be located at the very beginning,
containing a mapping table between offsets on the real disk and the
corresponding offset within the test file. Following the meta-data the
relevant sectors from the disk are just added (in any order as long as
the mapping table is correct).

Example on creating such a test file from a disk with only primary
partitions:

(echo "parted-test-file-v1"; \
echo "device-name=/dev/hda"; \
echo "device-size=40007761920"; \
echo "#disk offset=test file offset"; \
echo "0=512"; \
dd if=/dev/zero bs=512 count=1) | dd bs=512 count=1 of=tmp1
dd if=/dev/hda bs=512 count=1 of=tmp2
cat tmp1 tmp2 > hda.partedtest

This way, the only special treatment needed when processing test files
is the disk offset seeking (as well as a different length initialization
than "dev->length = dev_stat.st_size / 512;" in init_file in
libparted/linux.c). So in case of reading from a test file (as opposed a
real device) instead of seeking to the given sector, look up the sector
in a mapping table/hash table and seek to the return value.

In the example above only the partition table is considered. Parted
typically also read from within the file systems but that should not
be a problem, just add those relevant sectors as well. In fact I think
there should be a "record test file" command line option where parted
creates a test file filled with all the sectors it read from the disk.

SUPER BONUS:

With such "record test file" option, debugging problems reported from
users could be much easier to handle if the bug reporter could create
a test file from his or her disk that the parted developers then could
continue testing on themselves!

A typical bug reporting pattern today is that a user reports an assertion
error, but the issue bounces back to the user with a request of additional
information like "fdisk -l" output or similar.

It would be very nice to avoid this if the user could create a test file
containing all the necessary information at the very beginning and attach
that to the initial bug report.

BR Håkon Løvdal