Re: [lmi] VCS caching

[Greg Chicares]

On 2018-04-13 17:56, Vadim Zeitlin wrote:
> On Fri, 13 Apr 2018 16:12:41 +0000 Greg Chicares <address@hidden> wrote:
[...]
> GC> So git-worktree, and the worktree simulation above, would break your
> GC> use case.
> 
>  Yes.
> 
> GC> Similarly, it wouldn't work for Kim if for some reason she placed her
> GC> cache directory on another drive.
> 
>  I'm not sure why is it important for the directories to be on the same
> drive?

Same drive, different drives--it shouldn't matter at all.

[At this point in the antecedent email, I should have said that we must
abandon the git-worktree idea for the reasons above, and made it clear
that I wanted to introduce a completely different subject. Now that we
have settled on some small modifications to  'install_wx.sh', let's step
back and discuss how it should be invoked by 'install_msw.sh', so that
the MSW script calls the WX script in a way that works for all of us.

> GC> How do we make this work robustly in 'install_msw.sh', which says only
> GC>   ./install_wx.sh
> GC> today? I don't know where your LAN repository is, so I don't know how
> GC> to detect it

[...shell code to determine whether $wx_git_url is on a local drive...]

>  Of course, all this still assumes that it's actually worth to do all this
> instead of just cloning using --shared and personally I don't think so...

Good: let's not use git-worktree; instead, let's just clone.

I imagine you might run the WX script by overriding $wx_git_url, e.g.:
  $ wx_git_url='/my/lan/directory' install_wx.sh
But I'd like you to be able to run the MSW script without unpleasant
surprises like having it download 300 MiB from github that you had
already mirrored locally. Right now the MSW script includes the line:
  ./install_wx.sh
and if you run the MSW script right now, I think you will get that
gigantic download because of the way it invokes the WX script.

Please review commit c253ef10241, which I'll inline below to make it
easier to comment. I think that if you use something like a mount to
make the above usage
  $ wx_git_url='/my/lan/directory' install_wx.sh
translate to
  $ wx_git_url='/cache_for_lmi/vcs' install_wx.sh
then, with this change, the MSW script should DTRT for you.

I'm especially interested in your thoughts on new submodules that
might be added later--see the comments embedded in the change below.
Already I'm having second thoughts: it's trivial to copy a submodule's
.git directory to the cache directory, but it's not so trivial to find
that .git directory if you don't already know where it is.

---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
commit c253ef10241c0be835429023535bb44067aec3d3 (HEAD -> master)
Author: Gregory W. Chicares <address@hidden>
Date:   2018-04-13T22:39:46+00:00

    Use cached local wx git repository if found
    
    This is a request for review. It has been tested, so it's okay for
    production, but there may be better ways to accomplish the purpose.

diff --git a/install_wx.sh b/install_wx.sh
index 8922ac30..fe518ef6 100755
--- a/install_wx.sh
+++ b/install_wx.sh
@@ -23,9 +23,36 @@
 
 set -vxe
 
+# Preliminarily check local cache 
##############################################
+
+# Repositories are kept in /cache_for_lmi/vcs/ whence they can be
+# cloned cheaply and reliably--whereas cloning them from a remote
+# host takes considerable time and bandwidth, and fails if internet
+# connectivity is lost, or the host is temporarily unavailable, or
+# it is blocked by a corporate firewall.
+
+wx_repository="wxWidgets.git"
+default_server="https://github.com/wxWidgets";
+default_url="$default_server"/"$wx_repository"
+
+cache_dir="/cache_for_lmi/vcs"
+mkdir --parents "$cache_dir"
+
+if git ls-remote "$cache_dir"/"$wx_repository" >/dev/null
+then
+    default_url="$cache_dir"/"$wx_repository"
+    # Now we could make a list of submodules...
+    cd "$default_url"
+    git show master:.gitmodules |sed -e'/\.git$/!d' -e's!.*\/!!'
+    # ...and download any that are missing to local cache, but
+    # that seems awfully fancy. Instead, we'll let the code that
+    # follows download them to our clone, and if we're mindful,
+    # we'll move them from there to the cache.
+fi
+
 # Configurable settings 
########################################################
 
-wx_git_url=${wx_git_url:-"https://github.com/wxWidgets/wxWidgets.git"}
+wx_git_url=${wx_git_url:-"$default_url"}
 
 wx_commit_sha=${wx_commit_sha:-"e38866d3a603f600f87016458260f73593627348"}
 
--------->8-------->8-------->8-------->8-------->8-------->8-------->8-------