[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

March update on

From: Christopher Baines
Subject: March update on
Date: Thu, 16 Mar 2023 19:29:30 +0000
User-agent: mu4e 1.8.13; emacs 28.2


The last update was sent out back in January [1], while things are
stable there has been some activity in the last couple of months.


## Numbers currently provides ~2.2 million nars, which take
up ~9.2TiB to store.

Substitute availability is still reasonable, although recent
availability for i686-linux has been lower than it has been previously.

## Mirrors

Still no progress has happened in terms of mirrors. This would still be
nice to get setup, but it's probably not the priority in terms of
administration and infrastructure.

## Serving fixed output files by hash is now in the list of content addressed mirrors


In addition to the above change, some fixed output derivations use
download-nar rather than content addressed mirrors, so I've now updated
that to also fetch from [3].


## Referential integrity

This is usually something I think about in the context of databases, but
nars/narinfos have references, and I've been working to ensure that for
any nar provided by, you're also able to fetch any
nar it references.

I noticed something was lacking here a while ago when I went to
substitute an output for guix pull, but guix wasn't able to since some
referenced outputs weren't available. These missing outputs turned out
to be files that guix inserts in to the store to use as sources in the
derivations computed by guix pull. The build coordinator now knows about
this and will publish these as nars by default [4]. This was never an
issue when actually using guix pull, since it would compute the
derivations and create the missing outputs locally, so they aren't
usually substituted. To backfill the missing items, I wrote a script to
fetch the nars from, since it has them as part of
providing substitutes for the derivations.


There's still a bit more work to do in terms of ensuring outputs aren't
missing when nars are inserted and removed, but this is nearly there.

## Offering zstd compressed nars uses lzip to compress the nars, which is good
choice in terms of minimising the storage requirements and size of
downloads. It's been known for a while though that this might not suit
all users, as those with fast network connections may benefit from nars
that can be decompressed much quicker even if there's more bytes to


To enable providing zstd compressed nars, without having to store every
nar as an lzip compressed file and zstd compressed file, the nar-herder
now supports generating cached nars with different compressions.

The nar-herder running on bayfront (which serves
is now configured to generate and cache zstd nars [6]. Currently there
are ~25,000 cached nars taking up about 86GiB of space.


## Next steps

Last time I said that I thought the main blocking issue in making the default substitute server to try first was the
lack of zstd nars. Some nars are now available with zstd compression, so
there's progress been made on this at least. I think a good argument can
be made for trying being better for users than
trying first, but it's not clear cut.

On the issue of expanding the storage for nars, I've submitted the
request to purchase an additional hard drive for hatysa to expand the
storage there. There is still a need to come up with a plan to replace
bishan when it runs out of space (ideally before!).

Now that is building things for the master branch,
plus is submitting lots of builds for patches and
branches, there's more going on at varying priorities. This is really
good, but there's a greater need now to expose what's being built and
what the backlog looks like.

If you're interested in working on any of this, do let me know as while
I don't have time to work on much of it myself, I should be able to make
time to help others.

Let me know if you have any comments or questions!



Attachment: signature.asc
Description: PGP signature

reply via email to

[Prev in Thread] Current Thread [Next in Thread]