[bug#43160] Validate the result of our linux-libre sources clean up

[Mark H Weaver]

Hi Maxim,

Thanks again for the patches that you've already pushed.
They are a great improvement.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> I'm opposed to it because it would make it prohibitively difficult to
>> push micro kernel updates (most of which contain potential security
>> fixes) before Linux-libre has published their tarball release.
>
> Following recent discussions, I had understood that you agreed to wait
> for the Linux-libre releases before bumping our own releases.

Sorry, but that's incorrect.  I agreed to either wait or to look for new
blobs myself.  I've already exercised that second option a couple of
times since making that pledge.  I've found it to be quite easy.  The
overwhelming majority of commits to the stable branches are bug fixes
that obviously do not add anything resembling a blob.

If you haven't already done so, I'd encourage you to look over the
upstream commits between two consecutive upstream stable releases, to
see more concretely what I'm talking about, and how straightforward it
is to conclude that no new blobs could possibly have been added.

> It seems the Linux-libre releases occur fast enough to not pose much
> of a security issue; below is what I did to arrive to this conclusion.

The timestamp data you collected is clearly off by several hours.  It
appears to show that over two thirds of linux-libre releases come out
*before* the corresponding linux release.  One third of them appear to
have come out more than 4 hours before the corresponding linux release.
That cannot be right.  I guess someone's clock is off by several hours,
or somehow the timezones are not being taken into account.

I can tell you that within the last couple of weeks, since my pledge to
either wait for linux-libre or to check for blobs myself, I very clearly
remember upstream updates being tagged a few hours after midnight one
morning, and that it was not until after dark the next evening before
linux-libre had tagged their releases.  That instance is not reflected
in the data you collected, which again suggests to me that there's a
very significant systemic error in that data.

Even with this apparently large bias in the data, it shows that
linux-libre-5.7.7 was tagged almost 22 hours after the corresponding
linux release.  Most likely that was actually at least 28 hours.

Please don't misunderstand me: I do not fault the Linux-libre developers
for not being quick enough.  On the contrary, they've been consistently
*excellent* in that regard for as long as I've been paying attention.  I
certainly don't fault them for occasionally spending some time away from
their computers.

What I _do_ fault them for is *insisting* on placing themselves in the
middle of what should be a fast path for security updates from the
upstream developers to us.

To my mind, the practices that the Linux-libre developers are attempting
to impose on us feel like Service as a Software Substitute (SaaSS), but
in this case enforced by social pressure (and the suggestion that it
might violate the GNU FSDG) instead of the usual technical restrictions.

I'll also note that what you're proposing will apparently not be enough
to satisfy Jason and Alexandre.  They've gone so far as to suggest that
it's improper for an FSDG-compliant distribution to ever download
non-FSDG-compliant source code to a user machine.  It seems that to
satisfy them, Guix developers would apparently need to host our own
distribution site for cleaned-up copies of source tarballs, and to use
some separate tools to produce and upload new source tarballs to this
site for every update of software whose upstream source is not
FSDG-compliant.

I don't know about you, but I find their demands *oppressive*.

>> also make it prohibitively difficult to perform deblobbed bisections
>> between two adjacent versions from the upstream stable git repository.
>
> In my opinion, we should not trade our correctness guarantee in exchange
> for convenience,

First, I think that it's a mistake to suggest that any "correctness
guarantee" exists or could exist in Guix, with or without your proposed
patch.  Massive amounts of new code are flowing into Guix from upstream
projects on a daily basis, almost none of which is checked for
FSDG-compliance ahead of time.  It is widely acknowledged, even in the
FSDG document itself, that the most we can realistically hope to do is
to pledge to fix FSDG-compliance problems in a timely fashion if they
are brought to our attention.

Secondly, I think you're exaggerating the remaining risk.

I acknowledge that before these discussions began, the risk of
introducing new blobs was as high as 3% per Linux-libre update, which I
agree was too high.  However, we've made several important changes since
then.  Most importantly, I pledged to either wait for Linux-libre
updates or to manually check for new blobs, and you introduced a
'deblob-check' pass in 'make-linux-libre-source'.  Leo also made changes
to eliminate the risk of old deblob scripts being accidentally used.

It seems to me that these changes already reduce the risk of
accidentally introducing new blobs in our Linux-libre packages to near
zero, and probably at least an order of magnitude less than the risk of
non-FSDG-compliant code being introduced in, e.g., ungoogled-chromium.

> It'd be oversimplifying to say that the Linux-libre developers just
> run their scripts to produce a release; they also manually screen the
> new upstream changes and update their scripts accordingly.

Agreed, and we now account for that by either (1) waiting for them to
certify a new release or (2) checking manually for blobs ourselves.

> To give due credit to their efforts, we should not simply run their
> scripts with a newer version/commit of Linux and expect arriving at a
> correct result.

I've already agreed not to do that anymore.

It seems to me that some of these arguments are outdated, based on our
practices from before these discussions began.

>> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
>> 'make-linux-libre-source' should optional.
>
> Perhaps, like for the change proposed by Leo, the edge case of bisecting
> per-commit could be accommodated by reverting this patch when needed?

That can be done easily for Leo's patch, but not for yours.

In the case of Leo's patch, if I choose to manually check for new blobs,
I can simply change one line in a 'deblob-script-X.Y' definition, like
this:

--8<---------------cut here---------------start------------->8---
 (define-public linux-libre-5.8-version "5.8.7")
 (define deblob-scripts-5.8
   (linux-libre-deblob-scripts
-   linux-libre-5.8-version
+   "5.8.6"
    (base32 "07z7sglyrfh0706icqqf3shadf638pvyid9386r661ds5lbsa2mw")
    (base32 "0j6jba5fcddqlb42f95gjl78jisfla4nswqila074gglcrbnl9q7")))
--8<---------------cut here---------------end--------------->8---

In contrast, your patch changes the 'make-linux-libre-source' procedure
to *require* an existing 'linux-libre' tarball that precisely matches
what it's going to produce.  In other words, it removes the ability to
produce a new tarball, and has been reduced to merely being a verifier
of pre-existing tarballs.

Reverting those changes would not only be extremely invasive for a
simple micro kernel update, but would also, as a side effect, entail
redeblobing and rebuilding *every* version of linux-libre in Guix, even
if only one or two of the kernels needed updates.

I guess from my perspective, I see a lot of disadvantages:
disempowerment of users, occasional unnecessary delays on the order of
tens of hours to deploy security updates, not to mention having to do
another expensive git checkout and comparison on every kernel build, and
for what?

The main argument in favor, namely to reduce the risk of blobs being
accidentally included in our kernel updates, seems to be adequately
addressed by (1) my pledge to either wait or to check for blobs myself,
and (2) the recently-added 'deblob-check' invocation in
'make-linux-libre-source'.  Do you think these are insufficient?

Thanks again for this discussion, and for the work you've already done
to improve our deblobbing.

       Regards,
         Mark