Re: Solstice infrastructure hackathon

[Ludovic Courtès]

Hello Guix!

Ludovic Courtès <ludo@gnu.org> skribis:

> Here are tasks that were brought up:
>
>   • Set up a backup server for berlin.guix.gnu.org, the head node of the
>     ci.guix.gnu.org, possibly moving some services such as the web site
>     there.
>
>   • Add DNS redundancy for guix.gnu.org so it can point to one of two
>     hosts (need to figure out certbot challenges so both machines can
>     update their certificates).
>
>   • Set up status.guix.gnu.org with sysadmin status updates (possibly
>     using Prometheus?).
>
>   • Come up with a plan to add disks to the RAID array on bayfront, the
>     head node of bordeaux.guix.gnu.org.
>
>   • Work on a plan to back up the Disarchive database currently on
>     berlin.guix.
>
>   • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
>     plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>.
>
>   • Have a documented procedure to set up substitute mirrors, such as
>     the one in .cn (I can’t find the URL), ideally with plain rsync
>     access.

A small but dedicated bunch of people made progress on several of these
items today, in a loosely coordinated fashion on IRC—which perhaps made
it hard to get started, let us know what you think would help you join!

Most of the progress so far is visible in the commit log of the
maintenance repository:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/log/?id=e19f6d92b0b9a743c5b3cad236e51b8dd9d7c5e9

There’s IPv6, use of nar-herder to distribute bordeaux.guix substitutes,
I/O performance testing on a possible ci.guix head node
replacement/backup, web site replication, backups over rsync from berlin
to bordeaux, and more.

We have yet to complete support for web site replication: adding nginx
rules on the backup, having guix.gnu.org point to the two hosts, setting
up Let’s Encrypt.  This should be within reach quickly.

Other items above are yet to be addressed.  Our next priority should be
to have an off-site copy of the ci.guix substitutes.

Overall I think we need to aim for complete redundancy of the main
services.  The good news is that this Guix System thing greatly
simplifies the work!

To be continued with a second session sometime in January!

Thanks,
Ludo’.