Re: Inverted index to accelerate guix package search

[zimoun]

Hi Giovanni,

On Wed, 15 Jan 2020 at 12:53, Giovanni Biscuolo <address@hidden> wrote:

> zimoun <address@hidden> writes:

> > If Guix goes to Xapian, all the Git history of all the packages should
> > be indexed. Today, it is really hard to find the right commit
> > corresponding to the right version -- the only solution I have is "git
> > log"+grep; not really friendly.
>
> indexing the all the history could be very interesting, but it's enough
> interesting to me having a system to query (and tag if needed) Guix info
> from a point in time, i.e. index creation

Currently, "guix search" already does a good job (modulo the scoring
already discussed in length :-)).
If you read the doc about Guile regexp and recutils then you can query
(almost) all you want; or explain what you are not able to query. :-)

However, the current implementation cannot scale for all the packages;
other said find removed packages, or old version. And it is very
frustrating, especially when you use "guix time-machine".

A concrete example is given here [1] and "git log --grep" done in [2]
is not convenient at all. Another concrete example, see point 1. in
[3].

[1] https://lists.gnu.org/archive/html/help-guix/2019-06/msg00094.html
[2] https://lists.gnu.org/archive/html/help-guix/2019-06/msg00098.html
[3] https://lists.gnu.org/archive/html/help-guix/2020-01/msg00087.html



> thinking about implementation, IMHO indexing the output of "guix search"
> is doable but indexing the commit logs of git is complex, probably too
> much complex

I do not want to index the commit log message. But today, AFAIK, use
these commit logs is the only way to find the commit providing the
version an user want. And it is not cool, IMHO.

The whishlist is: be able to search through all the packages you can
manipulate with Guix. Other said index all the packages after the big
overhaul (inferior).


(Note that we are talking about packages but it is the same thing for services.)




> me too, as I said I'd like something like notmuch - guixmuch :-) - that
> indexes my mailbox; in Guix terms I see "my mailbox" [1] as
> packages/services/machines and all other useful metadata I have in all
> my channels/profiles

What do you mean by "other useful metadata"?


> ...then I have muchsync to sync the notmuch database across different
> machines, and a similar feature whould be nice for guixmuch :-O

It is already possible using manifests.



> [1] where IMAP is replaced by git (in various repos, for
> packages/config/channels) and offlineimap is replaced by guix pull

What you want already exist: manifest. :-)
And "guix publish" to expose your own substitutes if the package is
not build in the Guix farm and you have already built it.



> > However, note that index all the packages of all the Guix history
> > using guile-git+fold-packages is not straightforward: resource
> > consuming and piece of well-thought data structures.
> >
> > Well, now "guix time-machine" is here, Guix is lacking tools to browse
> > the history of all the packages.
>
> we are asking too much to Guix, but it is an interesting feature

I do not think so it is "too much". :-)


All the best,
simon