[bug#62264] [PATCH core-updates 0/6] Add `guix index` subcommand

[Antoine R. Dumont (@ardumont)]

Hello,

This introduces a new `guix index` subcommand (initiated by
civodul). This is in charge of indexing guix packages to ease user
lookups. There are 2 indexation methods:
- `manifest`: fast but less efficient, expected to be used by local
  users
- `store`: slow but more efficient, expected to be used on a guix build
  machine. Another service could be in charge of exposing a search
  package service to ease remote search of packages.

** Impact (positive)

tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file
search` cli we can find respectively in Nix or Debian's toolbox
(unbeknownst of their implementation).

This should help users transitioning from other distributions. Usually,
people will want to look up for files or packages they are already
using. They'd expect an easy way to find those back in guix. And we
currently cannot do so easily. We can ask on #guix-devel irc (as someone
mentions to me there) though that's not scalable nor really practical
for non-irc users.

** How did I test/run it?

I've been running both the:
- `guix index` subcommand to index packages (with both methods in
  separated or not dbs).
- `guix index search FILE` to actually search for packages present in
  the db

I've inspected time and again the db through the `sqlite3` program after
multiple runs too to ensure data were consistent. Data are consistent.

** Build?

The makefile routine will build that subcommand too.

** Development details

The backend technology is sqlite3. The command is in charge of migrating
the data model (if any). That data model migration process is
transparent for users. The data model is currently at version 3 because
it underwent some migrations during its development process
already. It's kept for dogfooding and examples reasons.

The --version flag mentions the default backend locations and data model
version:
--8<---------------cut here---------------end--------------->8---
$ guix index --version
Extension local cache database:
- path: /home/tony/.cache/guix/index/db.sqlite
- version: 3

guix index (GNU Guix) 1.4.0.3874-372b2
Copyright (C) 2023 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ git log | head -1
commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7
--8<---------------cut here---------------end--------------->8---

** Current caveat ("help needed on test")

The code works but...

I can't make the integration test i wrote work. Hence my delay to send
the overall patch (as the subcommand has been ready for a while). As
it's been way too long already, I've posted it in the hope someone
interested enough sees what's missing. Thanks in advance.

What's happening during the test is that somehow the execution of the
`guix index` subcommand triggers an exit code > 0 (error). Which,
expectedly, stops the test immediately. I don't get why the subcommand
exits that way and I don't see how to dig further unfortunately. Any
clues would be very much appreciated.

** Use

The subcommand is documented exhaustively.

--8<---------------cut here---------------end--------------->8---
$ guix index --help
Usage: guix index [OPTIONS...] [search FILE...]
Without argument, indexes (package, file) relationships from the machine.
This allows indexation with 2 methods, out of the local:

- manifests: This is the fastest implementation with the caveat of indexing
less packages. That'd be typically the use case of user local indexation.

- store: This is slowest implementation. It discusses with the store
daemon. That'd be typically the use case of building the largest db in one of
the build farm node.

With 'search FILE', search for packages installing FILE.

Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default.
See --db-path for customization.

The valid values for OPTIONS are:

  -h, --help          Display this help and exit
  -V, --version       Display version information and exit
  --db-path=DIR       Change default location of the cache db

  --method=METH       Change default indexation method. By default it uses the
                      local "manifests" (faster). It can also uses the local
                      "store" (slower, typically on the farm build ci).
The valid values for ARGS are:

  search FILE     Search for packages installing the FILE (from cache db)

  <EMPTY>         Without any argument, it index packages. This fills in the
                  db cache using whatever indexation method is defined.
Report bugs to: bug-guix@gnu.org.
GNU Guix home page: <https://www.gnu.org/software/guix/>
General help using Guix and GNU software: <https://guix.gnu.org/en/help/>
--8<---------------cut here---------------end--------------->8---

Example of an indexation (fast method by default):

--8<---------------cut here---------------end--------------->8---
$ guix index
Registering 133 packages  ▕█████▎                                             ▏
--8<---------------cut here---------------end--------------->8---

Example of a search:

--8<---------------cut here---------------end--------------->8---
$ guix index search sqlite3
sqlite@3.37.0        
/gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3
sqlite@3.36.0        
/gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3
--8<---------------cut here---------------end--------------->8---

Note: inspired from zimoun's way of quoting (/me like those, thx ;)

** Development process (optional read)

Note that this started as a subcommand (again from civodul's bootstrap
code).  Then as a good proposal from zimoun, this got simplified into a
guix extension (awesome work zimoun ;). Fwiw, developing it as an
extension felt way simpler for dev and run. Versus, having to setup my
machines to develop with guix. It continued as a guix subcommand in the
end as civodul proposed it that way instead.

Sent from my MUA as git-send-email would not work...
--8<---------------cut here---------------end--------------->8---

$ git send-email outgoing/0000-cover-letter.patch -a \
      --to=guix-patches@gnu.org
outgoing/0000-cover-letter.patch
Can't exec "sh": No such file or directory at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 231.
the editor exited uncleanly, aborting everything at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 252.
--8<---------------cut here---------------end--------------->8---

Hoping this finds you well nonetheless.

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8


===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch===
From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Sat, 18 Mar 2023 16:27:22 +0100
Subject: [PATCH core-updates 0/6] Add `guix index` subcommand
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

*** BLURB HERE ****

Antoine R. Dumont (@ardumont) (5):
  scripts-index: Transform `guix index` extension into a Guix script
  scripts-index: Store outputs alongside packages
  Makefile.am: Reference new script to compile
  Bootstrap tests for guix index subcommand
  Allow gcroot function to exceptionally ignore error

Ludovic Courtès (1):
  index: Add initial implementation from civodul

 Makefile.am            |   2 +
 guix/scripts/home.scm  |   2 +-
 guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++
 guix/store/roots.scm   |  10 +-
 tests/guix-index.sh    |  73 +++++
 tests/store-roots.scm  |   7 +-
 6 files changed, 686 insertions(+), 3 deletions(-)
 create mode 100644 guix/scripts/index.scm
 create mode 100755 tests/guix-index.sh


base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461
-- 
2.36.1

============================================================

signature.asc
Description: PGP signature