Re: [GNU-linux-libre] Good practices for removing nonfree code found in source code.

[Denis 'GNUtoo' Carikli]

On Sat, 02 Oct 2021 19:14:28 -0300
Alexandre Oliva <lxoliva@fsfla.org> wrote:
> The way GNU Linux-libre used to deal with it was to publish cleaned up
> tarballs, without history, and binaries built out of them.  More
> recently, we've started publishing a git repository with cleaned up
> sources, still without history: each one of our releases is an initial
> commit.
This looks really helpful for distributions that for one reason or
another depend on git. I should probably look into it for Replicant 11
at some point to make sure we don't distribute nonfree firmwares in
Linux. Though we'll probably modify it a bit not to block nonfree
firmwares. And we'll also add our patches on top.

> OTOH, the loss of commit history, though acceptable from the FSD
> compliance POV, is undesirable for various oher reasons.  So we're now
> working on a git repository rewrite that carries history but removes
> nonfree software and other objectionable bits from the point of
> introduction.  It's a parallel-universe history, in a way, but with
> git commit mapping, one could even use the rewritten repo for upstream
> development.
That looks really interesting. 

I also wonder how to have clean git commit mappings in cases like that.

The linux-libre use case looks even harder than what we have to deal
with in Replicant since there are so many versions of Linux and we
typically patch existing repositories that are already specific to an
upstream Android version (Like Android 6 or LineageOS 13).

Maybe some git features could be abused to do that. Git clone --depth
can have partial history, so we could probably reference commits that
don't exist (like linux commits) in linux-libre or some Replicant
repositories with that, but that isn't the full picture as even if we
abused that feature, but that doesn't really take into account the
scripts used to produce the difference between the free and nonfree
versions (like deblob / deblob-check) somehow.

For upstream development we probably also need to have good
relationship with upstream to ease something like that as there isn't
only the git repositories that are involved in contributing to upstream:
- When sending patches you are supposed to test the code you send.
  Since patches are sometimes rebased or moved from tree to tree, as
  I understand there is already some small changes being made between
  the git tree where the patch is being tested by the person who sent it
  and the tree where it's officially merged in the first time. So if
  the nonfree software don't affect the patches, it might be something
  upstream could accept.
- We probably also need to see how upstream react to people not using
  the nonfree firmwares in general. It will probably depends a lot on
  the group of people maintaining the drivers and the drivers.
  In some cases people even merged in patches to not depend anymore on
  nonfree firmwares and use a rom when the nonfree firmware is not
  found (like with some I.MX (6?) SDMA driver), in other cases I
  suspect that they expect users to actually run the nonfree firmwares
  (The Radeon driver bails out if the firmware is not found).

We'd also have to be able to handle trees like linux-next that are
rebased all the time and maintainers tree as well. If there isn't too
much manual work to be done in cases like that, it might be interesting
to be able to involve users as they might do that work for themselves
to use linux-libre with linux-next or some ARM maintainer tree for
instance.

> We haven't quite worked out how this is going to work, but the goal is
> that, as commits as added upstream, they can be incrementally verified
> and integrated, in a mostly automated fashion.
> 
> If/when we find a freedom bug in a previously-accepted commit, we'd
> have to re-rewrite that commit and all subsequent ones.
> 
> Given some script along the lines of deblob-check, the
> checking-and-integration is mostly automatable.  Once I grow up code
> for progressive integration of commits, I shall turn that into a free
> software project so that others can use it to maintain freed-history
> repositories of other blob-ridden projects.
A naive implementation could be that a more generic deblob script
could be converted to a git hook or set of git hooks to somehow rewrite
the history of different linux trees.

The hook would know what files are being changed so from that it could
trigger the operations on the files. It would also need to understand if
a file is being moved as git only has a notion of patches internally.

I've no idea if something like that could work for linux-libre or how
fast it could be as the logic probably evolves with the different linux
versions and files are being moved too, and that it would somehow need
to at least check a list of files for each commit.

If the code needs to be more generic than it actually is, there is also
coccinelle/spatch that may or may not work for that use case. It is
sometimes used in Linux to do tree wide patches that affect a huge
number of drivers and the advantage is that unlike regular expressions,
it does understand C.

Reading all the git documentation might also be a way to have more
efficient designs. For instance while looking a bit at it[1], the
documentation mentions https://github.com/newren/git-filter-repo to
rewrite git histories in a way that is way faster than what git already
has. Though I've no idea how to integrate a tool like that in a
workflow where a developer would clone a linux(-libre) repository.

Maybe it could look like some git plugin and the developer would do
that: 'git linux-libre clone git://[...]' or 'git clone
linux-libre://git://[...]' (instead of git svn clone or git clone
mediawiki://[...]).

Also about these suggestions, I don't know what linux-libre already
discussed or if what I suggested makes sense or not or is completely
obvious already.

> All this said, it doesn't seem to me that this platinum standard
> should be a requirement for FSDG compliance.  E.g., we don't seem to
> require projects to withdraw all past releases upon finding a freedom
> bug.  For compliance, fixing the bug and issuing updates with the fix
> appears to be enough to meet the requirements.  More than that,
> though IMHO desirable, appears to be beyond what could (should?) be
> considered as mandatory, given what's reasonably doable with existing
> tools.
Thanks

> Maybe with newer tools that make the process of cleaning history more
> convenient, we can eventually bump the requirements.
We'd also have to check that all uses cases are covered when doing that.

References:
-----------
[1]https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

Denis.

pgpQN_cNclWjG.pgp
Description: OpenPGP digital signature