SWH: extend sources.json and Mercurial (or not Git and not tarball)

[Simon Tournier]

Hi,

On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix is
>      irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

The file sources.json that SWH ingests only contains original upstream
and not our copies.  One step forward would be to also list the URL of
our tarball substitutes as the last mirror in sources.json.

Any taker? :-)

>
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
>      (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

Since most git-fetch origins use label tags, the one-to-one mapping is
not guarantee and we rely on SWH resolver using URL + label tag to get
the content from SWH.  For instance, if the label tag is changed
in-place by upstream pointing then to one different commit, then SWH
creates another snapshot but our fallback will fail (known issue:
history of history, etc.)

If we would have a list of identifiers instead of only NAR+SHA256, and
we could have Git commit ID here (or SWHID or others), then it would
ease the fallback machinery.

SWH folk is currently adding NAR hashes; they store it as ’ExtID’ (see
[1] and merge request [2]), but it is not clear yet how they would
expose the API entry point or if they would do.

Extending ’origin’ with another optional field using other
content-address keys would robustify the preservation of Guix.  Yeah,
indeed we could also build the X-to-SWH bridge with the Disarchive
database (global bridge) but it would appear to me better to have some
“local” origin-based bridge.

1: https://gitlab.softwareheritage.org/swh/meta/-/issues/4979
2: 
https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/merge_requests/459

Cheers,
simon