Re: generation of makefiles from build output?

[David Boyce]

Mark,

If one of your platforms is Linux I may have a good starting place for
you. I recently released an open source tool which does basically what
you want, though rather than simply looking at build output it
actually tracks file-access and process-creation events and is capable
of generating a Makefile from the resulting data. The idea is to be
able to do an audited build using the old system, then generate a
clean new build system from the collected data and deprecate the old
one. The website is http://audited-objects.sourceforge.net/.

Caveats:

- It has not operated at the scale you talk about. Among the
open-source packages I've used as test cases are PostgreSQL, OpenSSL,
and Apache HTTPD. For each of these, and a bunch more, it can generate
a new Makefile with complete dependency data which works right out of
the box. However these have a few thousand source files each, not a
few hundred thousand, so there could be scaling issues.

- I'm sure there are a few corner cases left.

- Most importantly, even in the case where everything works perfectly,
it's theoretically impossible to get back all the original flavor
because something is lost in the build. The task is analogous to
decompilation - the original build system may be capable of building
different product variants on multiple platforms,
debug/optimized/profiled/etc, while the audit-and-regenerate approach
can only give you a Makefile to reproduce the bits you actually built
and the rest must be grafted on manually. However, this can still have
significant value.

Speaking of Larry Wall, the Makefile generation part is a Perl script
so improvements should be a "simple matter of scripting" for a
reasonably experienced Perl programmer. In other words the auditor
itself is a C program which is complex, but once you have the audit
data for a build you can run the Perl script repeatedly to generate
test makefiles from it. There's lots of documentation but basically
you'd do

    % make clean
    % ao -o foo.ao make

to get a full audit in foo.ao, then

    % ao2make -o mf -full -members-only foo.ao

to generate a Makefile in "mf". Both programs spit out usage data with
a -help flag.

It's not an out-of-the-box solution, because makefile regeneration is
complicated, but if you want to solve your problem as described and
contribute to a general solution, this would be a good place to start.

-David Boyce

PS Ignore any talk on the website about the server side. There is a
server with a database for advanced uses but I'm not happy with it and
have plan to replace it when I have the time. In the meantime just use
the client standalone; it's simpler and it's the only part needed to
regenerate Makefiles.

On Sun, Jul 3, 2011 at 3:18 AM, Mark Galeck <address@hidden> wrote:
> Hello,
>
> I am working on a project which has more than 100M lines of code, more than 
> 100K
> files and, more than 1000 makefiles. Makefiles apparently have bugs (may build
> too little or too much), are slow, messy, and violate "Paul's rules" and 
> common
> decency on a routine basis.  I am asked to "do something".
>
> Well, seems that because of the size of the problem, fixing existing code is 
> not
> an option.
>
> I have an idea - get the full build output from scratch, and write Perl 
> scripts
> to process that output into well designed makefiles. It seems it should be
> possible to figure out, based on output, what the shell commands are that are
> used, what the arguments are, which arguments are options, which are files,
> which are generated from other shell commands, and so forth...  and figure out
> the makefiles from that.
>
> Has anybody heard of this kind of thing being tried on a large project? Is it
> hopeless? Do I not see some fundamental obstacle which makes it not possible 
> in
> practice? Perhaps somebody already wrote some scripts to do this kind of 
> thing?
>
> Thank you for any advice,
>
> Mark
>
> PS.  Larry Wall says good programmers are lazy. I am not good (not in Larry's
> intended meaning), but at least, I am really lazy (in his intended meaning) :)
>
> _______________________________________________
> Help-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/help-make
>