Can we speed it up? Prev: compiling guix is too slow?

[Pjotr Prins]

On Wed, Jul 05, 2017 at 08:49:52PM -0700, Chris Marusich wrote:
> "Feng Shu" <address@hidden> writes:
> 
> > Now I have found that 'guix pull' is too slow,
> > I need 3 hours to compile guix, is it possible to speed it ?
> 
> 3 hours is longer than it usually takes on my Lenovo X200 laptop with 2
> cores, an SSD, and 8 GB of RAM.  Usually, for me, it seems to take about
> 15 minutes.

Guix pull takes a while and building Guix in general is slow too. I
find I have to rebuild quite often (updating trunk, moving between
machines) and with having my own guix publish server building
Guix is the bottle neck(!) From our discussion on Friday I gather
it is going to get a lot worse with the growing number of packages. 
It scales probably worse than linear.

What will it be like with 15K packages? We will get there. We can
actually try it now by doubling the package tree - anyone wants to try
and create a simulation? I.e., not just double the tree, make sure
there are cross references between the two graphs by modifying some of
the inputs at random.

This leads to the following thought: why don't we create a 'lazy'
build. There are multiple ways to go about it (I think). One would be
to parse scheme files for package names and only compile those that
are needed when someone invokes a guix command (and have not been
compiled yet). Or generate a meta list for a source tree. Or
subcategorize packages so only those packages get included that are
asked for (assuming there are no deeper dependencies). For example,
few people need the bioinformatics packages. We could have the sub
section of the graph split out and have people do:

  guix package --topic=bio -i samtools

for example and compile the contents gnu/packages/bio/ directory when
that happens the first time for a specific checkout.

This would be a lot more scalable and the main package section could be
fairly small and build fast. 

Sectioning the graph may be hard (you'd be inclined to section off
languages and window managers), but I think it can be dictated by
whether a sub graph can live on its own. And if we allow packages to
reference packages in a sub graph (triggering the build) then anything
goes - the JVM subsystem could be in one and Perl CPAN in another.

I think scalability is a good goal and instant compilation another ;).
A few years back it just took 30 seconds to build Guix.

Pj.