Re: GNU make to consider files checksum

[Tim Murphy]

Hi :-)

2009/10/5  <address@hidden>:
>
> Tim Murphy wrote:
>> I think that checksumming might benefit some targets.  It would be
>> nice to be able to implement different "methods" for different targets
>> - because not all methods work well in all circumstances.
>
>> I have one example where every single file in a huge build includes 1
>> particular header file.  The file defines macros which are the
>> features that are enabled or disabled in the build.
>>
>> We know which features are used by particular components so in theory
>> we could work out not to rebuild components that are not influenced by
>> what's happened to the header file.  e.g. we could switch on a feature
>> or add a new feature without forcing a rebuild of the entire source
>> base.
>
> You can do that already today by simply splitting your global feature
> header file into smaller pieces and letting targets depend on only
> the relevant pieces rather than everything...

Yes, we have thought of that.  It's a good answer but it's hard to get
changes like that through into the absolutely enormous thing that it's
all used in.  That's just a human problem but it's more real than any
of the technical problems.  So it's not a solution we can use tomorrow
morning.

On the other hand what it the header file is stdio.h?  You can't
really do anything about that. what if the change is just a comment?

So I was searching for an example and that wasn't the greatest one.
What I really wanted to say was that for large files, an md5 checksum
is potentially a slow way to determine how out of date something is
but for small files it might be much more effective than a timestamp
and still be quick.

i.e. if you invent a new dependency mechanism then you need to be able
to balance where it is used so that it doesn't  end up making some
tasks worse.


>> This requires something like md5 but also some kind of "filter" to
>> determine what kinds of changes are significant to the particular
>> target that you are testing the dependency for
>
> IMNSHO, this is not a problem that make can (or should even attempt
> to) solve for you. This "filter" as you call it would have to know a lot
> about the the syntax your header and code files which makes it a bad
> candidate for a core make feature.

The filter would have to be external (i.e. not part of make) but it
could be much faster if it was a loadable plugin.

>> You can emulate md5 checksum dependencies  in make of course, using
>> temporary marker files, but it's a bit ugly and complicated..
>
> This problem is not strictly related to MD5 summing. With MD5 summing
> instead of timestamps, your global header file would still change and
> cause a full rebuild because this is what you explicitly asked for by saying
> that all targets depend on it.

One would md5 the filtered file (the result of the filter), not the
original.  One's filter would be on the features that affect the
current project.  So the filtered file would be unchanged even if you
changed the original and added new features as long as they weren't
ones that the current project (an exe,dll,lib,whatever) cared about.
This would mean that you need not rebuild any of the object files for
the current project.

So this would require a special-feature-filter-just-for-me.  GNU Make
wouldn't provide it but it might provide a way to load it and thus
make it fast enough to be worth using in a lot of places.

Arranging this in make as it is is complicated and messy because it
would involve creating temporary marker files that contained the md5
in their name.  This would lead to a mess of temporary files which it
would be hard to clean up precisely because by the time you want to
clean them you might not know what their real name is anymore.

We might do this one day without any help from make but I think it's
worthy for make to look beyond timestamps (actually beyond a lot of
stuff) and I am suggesting how.  I am constantly amazed by some of the
great features make has and I just think that a few more amazing
features wouldn't be a bad thing.

Regards,

Tim


-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/