bug#56398: (guix git) fails to check out repos with nested submodules

[André Batista]

Hi and I'm sorry for the delay..

sex 08 jul 2022 às 10:26:40 (1657286800), ludovic.courtes@inria.fr enviou:
> If we do this:
> 
> --8<---------------cut here---------------start------------->8---
> scheme@(guix git)> (define r (repository-open 
> "/home/ludo/.cache/guix/checkouts/4pe5bqkmsmcros5oviwzvmnjlie6jyhx2dciznz7shwawckb32sq/"))
> scheme@(guix git)> (define mod (submodule-lookup r "third-party/googletest"))
> scheme@(guix git)> mod
> $13 = #<git-submodule 17d1220>
> scheme@(guix git)> (submodule-update $13)
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> Git error: failed to resolve path 
> '/home/ludo/.cache/guix/checkouts/4pe5bqkmsmcros5oviwzvmnjlie6jyhx2dciznz7shwawckb32sq/third-party/googletest/.git':
>  No such file or directory

Nice trick!

> (...)
> access("/home/ludo/.cache/guix/checkouts/4pe5bqkmsmcros5oviwzvmnjlie6jyhx2dciznz7shwawckb32sq/third-party/googletest/.git",
>  F_OK) = -1 ENOENT (No such file or directory)
> (...)
>
> Thus, looking at ‘git_submodule_update’ in libgit2, it looks as if this
> condition was true:
> 
>   (submodule_status & GIT_SUBMODULE_STATUS_WD_UNINITIALIZED)
> 
> Hmm, thoughts?

Well, I guess ENOENT != GIT_ENOTFOUND and, in that case, I think we
are better off gracefully failing to update it than trying to "branch"
our local repo. The patch below tests for the directory's existence
before proceding with the update. With this patch applied I've managed
to refresh pytorch, its submodules and got the following code:

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/machine-learning.scm 
b/gnu/packages/machine-learning.scm
index b19af8a1d5..174ba3d39b 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -2871,7 +2871,7 @@ (define-public xnnpack
 (define-public python-pytorch
   (package
     (name "python-pytorch")
-    (version "1.12.0")
+    (version "82782")
     (source (origin
               (method git-fetch)
               (uri (git-reference
@@ -2881,7 +2881,7 @@ (define-public python-pytorch
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "0pdqi91qzgyx947zv4pw2fdj9vpqvdhfzw1ydjd4mpqm8g5njgnz"))
+                "0zshqfqv3lcwyym3l8zx675chnhpxn14c4nr1c2b7ci1zis785va"))
               (patches (search-patches "python-pytorch-system-libraries.patch"
                                        "python-pytorch-runpath.patch"))
               (modules '((guix build utils)))
--8<---------------cut here---------------end--------------->8---

Where "82782" is a ciflow/trunk reference to commit
700dba518be03ee0c0d6389162b5907a13838f49.

I've also thought of filtering the submodules list and passing the
resulting list instead, but came to conclude that it would clutter
the code instead of simplifying it.

I hope that helps!

---

>From 10bbb79f87f3728c347e33a101add8cb740e9469 Mon Sep 17 00:00:00 2001
In-Reply-To: <56398@debbugs.gnu.org>
References: <56398@debbugs.gnu.org>
From: =?UTF-8?q?Andr=C3=A9=20Batista?= <nandre@riseup.net>
Date: Thu, 4 Aug 2022 08:07:35 -0300
Subject: [PATCH] guix: git: Gracefully handle missing submodules when updating.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
To: debbugs@gnu.org

Fixes <https://issues.guix.gnu.org/56398>.
Reported by Ludovic Courtès <ludo@gnu.org>.

* guix/git.scm (update-submodules): Check if submodule directory
exists before trying to update it.
---
 guix/git.scm | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/guix/git.scm b/guix/git.scm
index 631bf577d3..d6a82fb86c 100644
--- a/guix/git.scm
+++ b/guix/git.scm
@@ -320,20 +320,22 @@ (define* (update-submodules repository
                             (fetch-options #f))
   "Update the submodules of REPOSITORY, a Git repository object."
   (for-each (lambda (name)
-              (let ((submodule (submodule-lookup repository name)))
+              (let* ((submodule (submodule-lookup repository name))
+                     (directory (string-append
+                                      (repository-working-directory repository)
+                                      "/" (submodule-path submodule))))
                 (format log-port (G_ "updating submodule '~a'...~%")
                         name)
-                (submodule-update submodule
-                                  #:fetch-options fetch-options)
-
-                ;; Recurse in SUBMODULE.
-                (let ((directory (string-append
-                                  (repository-working-directory repository)
-                                  "/" (submodule-path submodule))))
-                  (with-repository directory repository
-                    (update-submodules repository
-                                       #:fetch-options fetch-options
-                                       #:log-port log-port)))))
+                (if (file-exists? directory)
+                    ((lambda ()
+                       (submodule-update submodule
+                                         #:fetch-options fetch-options)
+
+                       ;; Recurse in SUBMODULE.
+                       (with-repository directory repository
+                         (update-submodules repository
+                                            #:fetch-options fetch-options
+                                            #:log-port log-port)))))))
             (repository-submodules repository)))
 
 (define-syntax-rule (false-if-git-not-found exp)
-- 
2.36.0