bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts.

[Janneke Nieuwenhuizen]

Janneke Nieuwenhuizen writes:

Hi!

> Ludovic Courtès writes:
>
> Hello!
>
>> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>>
>>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001
[..]
>> Anyhow, thanks for tackling this issue!
>
> Hehe.  You've probably seen Josselin's recent GraphML backend effort
> that might really help to address this?  I'm afraid this patch can maybe
> only postpone what really needs to be done...
>
> There is gc-stats output from a successful `guix pull' or `make
> as-derivation' on Guix/Hurd, that I can show you, and I've tried more
> than 20 times; it always fails (OOM, hang, spontaneous reset, ...).
>
> Below is a typical output of gc-stats on the Hurd for building self.scm,
> when heap-size peaks (using the the max 25 files patch):
>
> ((gc-time-taken . 1530)
>  (heap-size . 2,625,474,560)
>  (heap-free-size . 1127989248)
>  (heap-total-allocated . 1337029496)
>  (heap-allocated-since-gc . 28728)
>  (protected-objects . 28)
>  (gc-times . 324))
>
>
> notice that it's *much* bigger (more than twice) than my findings on
> linux-64 below.  I have no idea why this is of what it might mean...
>
> So I turned to Guix GNU/Linux to get some gc-stat measurements.  What
> you see below is the maximum head-size at any point (I also have
> heap-total-allocated but I think that's irrelevant? and initially didn't
> use a script that measured the time).
>
> * guix/self.scm: Vanilla, not chunked; print gc-stats.
> ((gc-time-taken . 27319485051)
>  (heap-size . 1,360,330,752)
>  (heap-free-size . 285,696,000)
>  (heap-total-allocated . 74,067,590,944)
>  (heap-allocated-since-gc . 186,250,144)
>  (protected-objects . 28)
>  (gc-times . 464))
> real  24m36.643s
>
> * guix/self.scm: Split building of directories into 26 chunks; print gc-stats.
>  (heap-size . 1,131,298,816)
>
> * guix/self.scm: Split building of directories into 26 chunks; no gc; print 
> gc-stats.
>  (heap-size . 1,121,116,160)
>
> * guix/self.scm: Chunks of 25 files; run gc; print gc-stats.
>  (heap-size . 1,066,725,376)
>
> * guix/self.scm: Chunks of 50 files; no gc; print gc-stats.
>  (heap-size . 1,299,230,720)
> real  26m40.708s
>
> * guix/self.scm: Chunks of 25 files; no gc; print gc-stats.
>  (heap-size . 1,024,045,056)  ; 1st run
> real  28m4.451s
>
> * guix/self.scm: Chunks of 10 files; no gc; print gc-stats.
>  (heap-size . 1,077,895,168)
> real  30m14.049s
>
> ...strangely enough, if we assume that these statistics translate to the
> Hurd, using chunks of max 25 files seems to be a sort of sweet spot?
> 25% less peak memory (~300MB), "only" 12% (3"45') slower...  though not
> great for GNU/Linux users...
>
> I have produced a handful of successful `guix pull's (from a local
> checked-out worktree) using the 26-way split and chunks of max-25 files
> patches, but sadly also many more attempts failed.  Initially, when
> creating this patch series, I was convinced this fixed building on the
> Hurd, but I'm much less enthusiastic now.
>
> So I still have a slight preference for using the latest max-25-files
> patch, but I'm sorry to say that I cannot back it up with tangible data.
> All in all a rather discouraging week with much effort spent for little
> gain.  Hopefully Josselin can do some of his magic here :)

Anyway, I've finally

--8<---------------cut here---------------start------------->8---
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=b2cc649999ed97b62517dcfb7cf21ee731c97487>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f2cfb4a85c82882c571274573fd7ff646d380b63>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=41df5c5289c970b40a002a32dcfdcb61ddc5a2f5>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=bd303443be97c5aa5f8696f6f1ecc5e40b919874>
--8<---------------cut here---------------end--------------->8---

managed to get all remaining necessary dependencies built to run `guix
system reconfigure' on the Hurd*.  The only patch that we need that's not
on master yet is this one.

I'm still rooting for Josselin's graphml cycle analysis and/or lechner
Felix's bespoke make rewrite will render this hack obsolete real soon
now...

If there are no objections I'll go forward and install this patch by the
end of the weekend.

Greetings,
Janneke

*) using the hurd-team branch, grub install still fails, apparently
   because "someone" imagines /dev/wd0 to be /dev/hd0, but the new
   system derivation is being enabled.

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com