bug#65391: Acknowledgement (People need to report failing builds even though we have ci.guix.gnu.org for that)

[Simon Tournier]

Hi Arne,

    ( I have not re-read all the thread. )

On Mon, 11 Sep 2023 at 10:30, "Dr. Arne Babenhauserheide" <arne_bab@web.de> 
wrote:

>> Well, I do not think that any policy will mark a package for removal on
>> the first build failure.  However, if the same package is still failing
>> after several X <duration> or attempts, it means something is wrong.
>> Marking it as a candidate for removal implies:
>>
>>  1. check if the failure is from CI when it builds locally,
>>  2. keep a set of packages that we know they are installable.

> This is a good example, but not for removing broken packages. For
> perl6-xml-writer removing the package would keep breakage in Guix.
>
> I just checked the build, and this looks like a Guix packaging error

This is exactly the effect if we have a policy. :-)

Please, do not read a policy for the removal of broken packages as an
automatic process.  As you, I think an automatic process for removing
would be a bad thing about the user experience.

Maybe I misunderstand what a policy is.  For me, a policy is a plan that
is used as a basis for making decisions, a policy helps in reaching
conclusion which then can lead to some actions.

Somehow this discussion is the implementation of the policy I am
proposing and that would help the maintenance, IMHO. I have manually
marked this package for removal and…

> that breaks the tests due to a change to some unrelated package:

…surprise, surprise, someone has checked. :-)

A policy for removal about the broken packages would allow to know what
to do.  If the same package is still failing after several X <duration>
or attempts, it means something is wrong.

Currently, either you hit a broken package when doing some Guix
operations.  And that is a very poor experience, IMHO.  Either one have
to open the dashboard from CI [1], select some red buttons and
investigate.  And we can count with few fingers the number of people
doing that.

1: https://ci.guix.gnu.org/eval/741273/dashboard


> Disabling the tests makes the package build and work.

Here is the point of my proposal to have a policy for removal of broken
packages: automatically check how many times they have failed to build
and automatically tag them when they are considered problematic.  If no
one care and these tagged packages are not fixed, then let remove them.

It would drastically help in the maintenance.  Otherwise, your help is
very welcome in monitoring all the failures. :-)


> So here, removing a package would start at the wrong place: some change
> between 2021-02-01 and 2021-04-30 broke the perl6-tap-harness and we did
> not detect that.

Yes, that’s where QA should help: detect unrelated change that have a
long distance impact on unrelated packages.

        Changes to the branching/commit policy
        Christopher Baines <mail@cbaines.net>
        Thu, 08 Jun 2023 15:24:37 +0100
        id:87y1kuyqew.fsf@cbaines.net
        https://yhetil.org/guix/87y1kuyqew.fsf@cbaines.net
        https://lists.gnu.org/archive/html/guix-devel/2023-06

        [bug#63459] [PATCH] doc: Rewrite the branching strategy.
        Christopher Baines <mail@cbaines.net>
        Fri, 12 May 2023 08:55:20 +0100
        
id:f339d15842370b97558b704593848e318462b68d.1683878120.git.mail@cbaines.net
        
https://yhetil.org/guix/f339d15842370b97558b704593848e318462b68d.1683878120.git.mail@cbaines.net
        
https://issues.guix.gnu.org/msgid/f339d15842370b97558b704593848e318462b68d.1683878120.git.mail@cbaines.net



> This does not mean that there will never be a case in which a package
> has to be removed, but given that both cases you showed are likely
> self-induced breakage due to changes that should have been rejected as
> breaking seemingly unrelated packages, it rather looks like the
> situation where removing the package is the right way forward is the
> exceptional case.

We are miscommunicating.  Or we have a very different vision about what
should be the reliability of Guix.

As a regular user, I need perl6-tap-harness, so I type “guix install
perl6-tap-harness”, and bang, it fails.

As a regular user, I do not mind if the problem is coming from some
change between 2021-02-01 and 2021-04-30 or if it comes from something
else.  What I want is that “guix install perl6-tap-harness” just works.

Having a clear policy for removal – again not an automatic removal
procedure – would help all, IMHO.


> The norm is that our CI should have detected a problem in the commit
> causing the breakage.
>
> Can we automatically rebuild all inheriting packages when a package gets
> changed?

CI builds all the commits pushed to Savannah.  Not exactly all but
that’s another story and it does not matter for this discussion.

AFAIK, no one is checking that the commit they are pushing does not lead
to break something.  Else they would not push it I guess. ;-)

Instead, it is QA that builds “pre-commit“ (patches).  Thanks to
tireless Chris’s work since years, we have some tools for monitoring the
impact of one change on the whole package set.  Somehow, if I have
correctly understood, QA uses the Build Coordinator to list all the
derivations and then build all the new ones generated by the change.

So the answer to your question is yes. :-) Aside, help is welcome for
improving QA.


> Also see above: in the two cases you selected, removing the package
> would be the wrong path forward.

Removing a package that is broken since 2021 is the good path forward.

If you care about one package that is marked to be removed soon, then
you fix it or raise your concern.  Else it means no one care and so what
is the point to keep broken packages that no one uses?


>> On Wed, 30 Aug 2023 at 12:39, "Dr. Arne Babenhauserheide" <arne_bab@web.de> 
>> wrote:
>>> If a change in packages breaks my manifest, that is extremely painful.
>>
>> Yeah, and such rule for dealing with broken packages will be helpful for
>> detecting such change and so avoid such situation.
>
> Since a manifest is strictly dependent on all packages defined in it,
> removing a single referenced package means that the manifest is broken:
> no update works anymore. No security updates come in anymore — even if
> the package in question worked locally. This is a situation we should
> not cause.

Again, I am not proposing an automatic removal process but a policy.  A
policy could imply some news or some message saying: these packages will
be removed soon because they are broken.

Assuming this case: the package fails on CI and pass on your machine.
Let assume you have not been enough annoyed for reporting the failure of
the substitutes.

Currently, the situation can stay like that for a long time.  It means
that each time something in the dependency graph of that package is
changed, then we burn electricity for re-building it for nothing.

What I am proposing is: if the same package is still failing after
several X <duration> or attempts, then we mark it as ‘broken’ and it
becomes a candidate for a removal.  People who care raise their hand.
And we have a better idea about the real status.


> The more important question is (serious question and *not* for assigning
> blame, but to see whether we can improve processes): with the time we
> already spent in this discussion, we could have fixed a lot of packages.

This was exactly what I was going to answer you. :-)

> Why did we not do that?

I speak for myself, for many packages that are broken, my first question
is: is it worth to investigate?  My estimate starts with a mix between
do I need them? and will the user experience be better compared to my
time spent to investigate.


Cheers,
simon