Re: versioning config.cache

[Dave Hart]

Eric Blake <address@hidden> wrote:
> According to Dave Hart on 5/1/2009 5:37 AM:
>> I wanted a
>> version number to bump that would cause configure to ignore
>> config.cache results from prior versions.
>
> Interesting concept.  However, I'm worried about one thing - it is also
> possible to invalidate the cache by updating external software (installing
> a newer library used by your package, or even upgrading the OS without
> corrupting the directory layout).  Thus, in addition to anything your
> package does, it is also possible that the cache can be wrong even without
> a change in your package.  I think it is a bit misleading to add
> auto-clearing if we can't reliably auto-clear in all situations.

Thanks for your timely and thoughtful reply.  I agree external
conditions can invalidate the cache as well, and I do not want to
suggest clearing it due to version mismatch will eliminate the need to
remove the cache manually.  That is the big trade-off with using it at
all, the first diagnostic step has to be to toss the cache and really
run configure.  That eats up some of the prior time savings with extra
testing.  I appreciate your concern that changing from the cache being
essentially completely operator-managed to more automated could give a
false impression that configure now knows how to clear the cache
itself whenever needed.

> One other thing to remember: invalidating the cache most often affects
> those who are tracking incremental development (snapshots), and not those
> tracking released tarballs (where each tarball expands to a new directory
> name, and thus does not share the cache with the previous version).  In
> other words, since the people most likely to be bit by invalid caches are
> already those who are brave enough to track bleeding edge, you can expect
> them to be able to understand and do more about the situation.
>
> What's wrong with telling your users that if they track incremental
> development, that they must run 'rm config.cache; ./config.status
> - --recheck' any time you commit a change that invalidates the cache from
> your end?

On one hand, there is nothing wrong with that approach and it remains
true even with version-stamping of config.cache.  However, I think
it's asking too much of our volunteer developers who are only very
loosely coordinated to know whenever a potentially incompatible change
has occurred.  In honesty, very few even want to think about Automake
and Autoconf, they are simply spoiled by the results and blissfully
ignorant of the details.  It seems useful to me for those few who do
maintain NTP's Autoconf to be able to automate avoidance of this class
of predictable problems.

> Or put another way, I'm not sure you've convinced me that
> autoconf needs to add this feature (although I have not outright rejected
> it either, so more discussion may still sway my opinion).  At any rate,
> thank you for an example macro that uses a different namespace than AC_; I
> wish more developers would do that.  And if it works for you, you can
> certainly continue to use it.

My experience in the Autoconf world is dreadfully limited.  I am not
attempting to convince you to add anything, particularly this as it's
quite green.  I've played with it for a few days and received help
with portable shell programming from Harlan Stenn as well as the
Autoconf manual, and now from Eric Blake.  Thank you.

> Some comments on your macro:
>
>>       case "$cache_file" in
>>        ../*)
>>           # Parent configure will have cleared cache once already.
>>           # This will misfire if a top-level configure is invoked
>>           # with --config-cache= value beginning with '../', is
>>           # there a better way to detect this configure was
>>           # invoked by a parent directory configure?
>>           ntp_cache_flush=0
>
> Not off the top of my head.  But remember, the only point of a cache is to
> speed things up; not using a cache will still get correct results although
> with more time.  So although it takes longer, it doesn't hurt to guess
> conservatively and get a wrong answer.

Before my version-based clearing, the first run time of our child
configure scripts was dramatically less with --config-cache, as nearly
everything in those smaller configure scripts was already tested by
the massive parent NTP configure.  If I did not make the ../*
exception, the sequence of false automatic cache clearing would make
me look silly.  With one parent and two child configure scripts,
building from a tarball would run the parent uncached (of course) the
first time, then each child in turn would throw away the
just-accumulated config.cache from the parent and start anew.  Only
after all three uncached runs of configure churned to completion,
repeating tests, would future configure runs use the cache.

Depending on $cache_file beginning with ../ only for children means I
can't have several build targets with different configure parameters
in subdirectories point to a shared config.cache-$hostname in the
parent directory (which also contains configure).  I have scripts that
solve this problem, but it seems like a more reliable test would be
useful
to determine if a configure script is top-level or subdir.

>>     case "$ntp_cache_flush" in
>
> Since this variable is under your complete control, I would have written
>
> if test $ntp_cache_flush = 1; then
>  ...
> fi
>
> rather than a case statement.

That's a fine choice.  I think for my purpose I still prefer the case
for legibiity, though i realize it's inefficient.

>>       c_varname_list=`set |
>>                       sed 's/=.*$//g' |
>
> Why the g modifier?  If you are deleting all characters starting at the
> first =, then there aren't any other matches for = left in the line, so
> the g will never match twice.

Because I'm lazy and did not think very much about it.  It's gone now.

>>                       fgrep _cv_ |
>
> fgrep is not portable (on some platforms, it MUST be spelled 'grep -F';
> see also how autoconf computes $FGREP, although it will dirty the cache if
> you call AC_PROG_FGREP this early).  Also, since you already used sed in
> this pipeline, you may want to consider rewriting this to do the entire
> filtering through a single sed script (sed can do pretty much anything
> that grep can; it is more expensive to fire up sed than grep, but once you
> already have to pay the cost of sed, it is cheaper to use one sed process
> than a sed|grep pipeline).

You are thinking about performance on a completely different plane
than me.  I'm impressed, and stunned, that anyone would seriously
consider shaving a couple of grep invocations from a
once-per-configure snippet of code given the obfuscation compared to
the more rudimentary pipeline.  Still, I will look at it, it's a fun
puzzle.

>>                       grep -v 'ntp_cv_.*_cache_version'
>>                      `
>>       for c_varname in $c_varname_list
>>       do
>>           dnl use AS_UNSET([$c_varname]) eventually
>>           eval ${c_varname}=;
>>           $as_unset $c_varname
>
> Why not use AS_UNSET right now?

NTP requires Autoconf 2.59, though in just the last few days there has
been discussion about raising that to 2.61.  I do not know which
version of Autoconf introduced AS_UNSET, but it was about 6 months
ago.  A few times now, I have found myself wishing the wonderful
Autoconf manual told me more often about the minimum version of
Autoconf required to use a particular macro.  I love the cleanup and
improvements going on in M4sh, Autoconf, and Automake, and would
nearly always prefer to be using the latest stable stuff for NTP.
Others have a different perspective and want the broadest convenience
in build host selection without always requiring such slavishly
up-to-date tools.

Thanks again for your helpful response.

Cheers,
Dave Hart