Micro releases and testsuite work (was: Re: [FYI] {micro} tests: remove some code duplication)

[Stefano Lattarini]

Hi Peter.

On 05/22/2013 06:35 PM, Peter Rosin wrote:
> On 2013-05-22 15:57, Stefano Lattarini wrote:
>> * t/ax/am-test-lib (null_install): New function.
>> * t/instdir-java.sh: Use it instead of copied & pasted code.
>> * t/instdir-lisp.sh: Likewise.
>> * t/instdir-ltlib.sh: Likewise.
>> * t/instdir-prog.sh: Likewise.
>> * t/instdir-python.sh: Likewise.
>> * t/instdir-texi.sh: Likewise.
>> * t/instdir.sh: Likewise.
>> * t/instdir2.sh: Likewise.
> 
> Hi!
> 
> Reading about micro releases in HACKING:
> 
> * Micro releases should be just bug-fixing releases; no new features
>   should be added, and ideally, only trivial bugs, recent regressions,
>   or documentation issues should be addressed by them.
> 
> Looking at this change (and possible others, I haven't checked
> closely) I don't see the fit.
>
Indeed, I think testsuite refactorings should be added to the list
above (see patch below). After all, causing possible disruption or
spurious failures in the testsuite is not nearly as serious as the
introduction of a regression or a backward-incompatibility.  In fact,
I'd even rather see such a disruption exposed early, in a micro release
(where we can be much more confident about the overall correctness and
health of the code base) than in a minor or major release, where it might
be more difficult to discriminate between issues caused by work on
the code and issues caused by work on the testsuite.

> Now, I'm not complaining, but the above wording gave me the impression
> that micro releases would contain very few commits, but it seems
> that commits to micro are a lot more frequent than I anticipated.
>
So far, apart from a minor bug fix, all the commits on 'micro' has
been testsuite-related.  That is how it should be IMHO.  But you are
right that this wasn't made clear at all in HACKING.

> The disconnect is perhaps that changes to tests are not explicitly
> mentioned?
>
And that they do not influence how Automake will work on real-world
packages.  That is, a testsuite regression is not going to annoy or
inconvenience Automake users, but only the Automake developers.

> And since "non-trivial but mostly safe" cleanups are
> allowed in minor releases, I would assume that trivial cleanups
> fit in micro as well?
>
Not really.  And in this case, the testsuite refactoring done in
'micro' so far are actually not very safe either (it took me a
while to write them in a way that left the testsuite passing on
FreeBSD, NetBSD and Solaris 10).  What makes them acceptable is
that they touch only the testsuite.

> Maybe that should be mentioned explicitly?
>
The patch below should make things clearer.  WDYT?

> Anyway, I don't really care, I'm just a bit surprised...
>
I hope thsi reply of mine clarifies things.

Regards,
  Stefano

---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ---- 8< ----

>From d9a3a4477bdc346f515786cf70568db25a167422 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Wed, 22 May 2013 20:13:41 +0200
Subject: [PATCH] HACKING: it's OK to do testsuite refactoring in a micro
 version

Reported-by: Peter Rosin <address@hidden>
Signed-off-by: Stefano Lattarini <address@hidden>
---
 HACKING | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/HACKING b/HACKING
index 3ce2e66..194a775 100644
--- a/HACKING
+++ b/HACKING
@@ -111,7 +111,10 @@

 * Micro releases should be just bug-fixing releases; no new features
   should be added, and ideally, only trivial bugs, recent regressions,
-  or documentation issues should be addressed by them.
+  or documentation issues should be addressed by them.  On the other
+  hand, it's OK to include testsuite work and even testsuite refactoring
+  in a micro version, since a regression there is not going to annoy or
+  inconvenience Automake users, but only the Automake developers.

 * Minor releases can introduce new "safe" features, do non-trivial but
   mostly safe code clean-ups, and even add new runtime warnings (rigorously
-- 
1.8.3.rc2