[bug#50960] [PATCH 10/10] shell: Maintain a profile cache.

[Ludovic Courtès]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op ma 04-10-2021 om 10:19 [+0200]:
>> > A documented flag to always consider the cache stale seems good, though I 
>> > think
>> > at least the dependencies made with the common macros and procedures 
>> > 'include',
>> > 'load', 'include-from-path', 'load-from-path', 'use-modules' and 
>> > non-recursive
>> > 'local-file' could be tracked, though this could be left as a TODO for 
>> > later
>> > I suppose.
>> 
>> Tracking those uses reliably is impossible: there could be same-named
>> bindings that do other things, there could be custom macros, there could
>> be “dynamic arguments” (whose value is not known statically), etc.  You
>> have to expand + evaluate the code to get better results, and even then,
>> there might be different paths in the code so you can’t be sure you got
>> it right.
>
> I think there's a miscommunication here.  From what I'm reading, what you have
> in mind is that, to determine the dependency information, "guix shell" would
> open "guix.scm", read it with the procedure 'read' and sniff for 'load',
> 'include-from-path', 'load-from-path', 'use-modules' and 'local-file' form
> -- something like 'module-file-dependencies' and 'extract-dependencies', but
> more general.

Yes, that’s roughly what I thought you had in mind, sorry!

> However, my idea is to replace these macros, such that, when "guix shell"
> loads "guix.scm" or "manifest.scm", these macros inform "guix shell"
> that "guix.scm" or "manifest.scm" depend on certain files referred to
> by 'load', 'include-from-path', etc. forms, using a mechanism like the
> 'notice-dependency' defined in <https://issues.guix.gnu.org/50384>.
>
> Then, when "guix shell" puts the resulting profile in the cache,
> it includes the generated list of files.  And when "guix shell" finds
> an entry in the cache, it will check if the files in this list (and guix.scm
> or manifest.scm of course) have been modified.  If some are (or the forcing
> flag is set), guix.scm needs to be loaded and a new profile needs to be
> generated.  If they are all unchanged, guix.scm will _not_ be read: the cached
> profile can be reused.
>
> It's not 100% reliable (e.g. the list of packages in the manifest could
> depend on the phase of the moon if (current-time) is used) (is that what
> you mean by ‘different paths’ and ‘dynamic arguments’?), but it should
> cover the vast majority of cases.
>
> I don't know a non-artifical situation where ‘custom macros’ are a problem
> -- do you know an example in the wild where this dependency tracking scheme
> would fail to work?

I think we have to look at the cost and at the benefits.  The cost:
either we spread ‘notice-dependency’ calls all over the place, or we
interpose on ‘open-file’ altogether, assuming that even works.  The
benefit: we’d have an 80% solution.  There would still be edge cases not
handled correctly; a realistic example is a ‘guix.scm’ whose execution
differs based on the presence of a file or environment variable.  People
would have to know about this pitfall and pass the right flag.

[...]

>> In such situations, I err on the side of not even trying.  The added
>> complexity for a flaky result doesn’t pay off to me.  I prefer to be
>> upfront, document limitations, and let users handle them as they see
>> fit.
>
> About complexity: there's some extra code, sure, but it doesn't seem complex
> to me.  To track dependencies in <https://issues.guix.gnu.org/50384>,
> I only needed to add 'notice-dependency' and some other code to (guix build 
> compile),
> and some code to build-aux/compile-all.scm to save/load the dependency 
> information
> to/from the builddir and to also check the mtime of dependencies.
>
> Does it still seem flaky to you, after my explanation on how the dependency
> information would be determined?
>
> Being upfront, documenting limitations, and having a ‘force rebuild flag’
> (is that what you mean by ‘letting users handle them as they see fit’?)
> (possibly also documenting 'notice-dependency') is not incompatible with
> the automatic dependency tracking.
>
> More abstractly, this seems more like a ‘Perfect is the enemy of the good’ 
> situation.
> While I would prefer 'perfect' above 'good', and the automatic dependency 
> tracking
> certainly isn't 'perfect', it does seem 'good' to me, and perfection appears 
> to
> be impossible and there's the ‘force rebuild flag’ as an escape hatch, so I 
> believe
> 'good-but-not-perfect' to be acceptable here, as long as it is documented in 
> the manual
> that there are some limitation on what dependencies are automatically tracked.

Yeah, I can relate to that sentiment, even if I’m all too often on the
“perfectionist” side of things.  ;-)

I guess I could live with an 80% dependency tracking solution.  However,
my criteria for it would be that (1) it should be orthogonal (e.g., not
requiring explicit calls to ‘notice-dependency’ in many places), and (2)
it should be self-contained and reasonably small.  Perhaps having
‘load*’ or similar instrument ‘open-file’ or similar could help achieve
that?

I would rather not block ‘guix shell’ on that though.

WDYT?

Thanks for taking the time to clarify what you had in mind!

Ludo’.