Re: RFC: libgit2 is slow/inefficient; switch to git command?

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> While attempting to bisect against the Linux kernel tree, the
> performance of libgit2 quickly became problematic, to the point where
> simply cloning the repo became a multiple hours affair, using upward to
> 3 GiB of RAM for the clone and indexing of the objects (!)

Did you confirm with a pure Guile-Git snippet that calls ‘clone’ that
this is the behavior observed?

> Given that:
>
> * the git CLI doesn't suffer from such poor performance;
> * This kind of performance problem has been known for years in libgit2
>   [0] with no fix in sight;

This reports talks about 5x wall-clock time, which is obviously not
great, but it doesn’t talk about memory usage, does it?

It talks about SHAttered though; that’s a key consideration to make sure
we’re doing an apples-to-apples comparison.

> * other projects such as Cargo support using the git CLI and that
>   projects are using it for that reason [1];

Should we follow Cargo’s lead for packaging as well?  :-)

> Would it make sense to switch to use the git command directly instead of
> calling into this libgit2 C library that ends up being slower?  It would
> provide a hefty speed-up when using 'guix refresh' or building new
> packages fetched from git without substitutes, or using 'git-checkout',
> etc.
>
> What do you think?

I think that’s not an option.  The level of integration we have in (guix
git), (guix channels), etc. is not achievable by shelling out to ‘git’.

"Philip McGrath" <philip@philipmcgrath.com> skribis:

> Along those lines, there’s an implementation of clone/checkout in pure Racket 
> (for the package manager) that could probably be ported to Guile relatively 
> easily. I’d expect libgit2 to be faster for the things that it supports, but 
> the Racket implementation does support shallow checkout, so it might pay off 
> if that skips a lot of work.
>
> Code: 
> https://github.com/racket/racket/blob/master/racket/collects/net/git-checkout.rkt
> Docs: https://docs.racket-lang.org/net/git-checkout.html

That sounds like a worthy avenue; support for shallow clones would
already be an improvement.

> (More broadly, I haven’t investigated performance issues, but my basic 
> inclination would be toward improving libgit2 over running the git 
> executable.)

Same here.  The way I see it, we could gradually move bits of Guile-Git
to being pure Scheme.  So perhaps the first step would be to provide a
pure Scheme ‘clone’ based on the Racket code above?

Thanks,
Ludo’.