Re: Re-write of documentation tree build process

[John Mandereau]

Jan Nieuwenhuizen wrote:
> John Mandereau <address@hidden> writes:
> 
> > Should I send a patch again, or work according to your remarks and push
> > to stable/2.10 (and possibly master) if Jan agrees?
> 
> Please push to stable/2.10.

Just done. I've spent so much time testing and debugging that I think
stable/2.10 is almost ready for a Christmas release :-)



> > - we don't add any custom header, so all stuff using the "header"
> > variable could be dropped. 
> 
> If it's not too much of a mess, keep the header stuff in.  We might
> want to add a header.

Ok. But I've scrapped remove_self_ref() because no HTML file in the docs
currently needs it. @WEB-TITLE@ substitution isn't used either, but
since it was less messy, I kept it.


> > I've already experienced that '[.]' is bogus in some cases.
> 
> I'm not aware of that?

Sorry, I was mistaken. There was a bug in my code not because of a '[.]'
but because of an unescaped '.' :-P


> > Does the 'EN' means that the page language is English? If so, it should
> > be modified to handle pages of any language.
> 
> Yes, I think so.

After what Juergen wrote, it's safer not to change it.



> I have two remarks, but given the state add-html-footer was in, I
> hardly feel I can make any demands; you're free to ignore me.

You remarks are very interesting, especially towards a programming
newbie like me, but I already spent a lot of time on this... IMHO,
programs like these buildscripts won't grow to thousands of lines of
code, so as long as they work, they can bear not totally perfect code.
Anyway, I take your remarks as general programming rules :-)


> +mirrortree.hardlink_tree (input_roots = doc_dirs,
> +                          process_dirs = outdir,
> +                          strip_dir_names = outdir,
> +                          exclude_dirs = r'(fr|po|out|\w*?-root)(/|$)',
> +                          process_files = 
> '.*[.](?:midi|pdf|png|txt|ly|signature)$|VERSION',
> +                          exclude_files = 'lily-[0-9a-f]+.*.pdf',
> +                          target_pattern = target_pattern,
> +                          targets = targets)
> 
> in general, if a function takes much more 3 parameters, you may want
> to take a second look why that is.  It may be to be too generic, or
> perhaps it tries to do too many things and can be split into several
> functions, or some parameters can more naturally be grouped in a
> class.

hardlink_tree is actually very generic and tries to do many things :
copying a tree of directories, hardlink files, find files and copy
symlinks, all that with 'out-www/' stripping. 

The function takes 9 parameters, because I wrote it like a bunch of
shell commands within a makefile, with a bunch of variables. I tried to
make this awful function call a bit more readable with keywords
arguments.

If I really wanted to rewrite it, I'd split it into:
- a function that walks the input directories and returns directories
and files lists
- filtering these lists
- mapping the lists with mkdir, hardlink and symlink standard library
functions


> IIRC Han-Wen suggested to write Python scripts, because the mess of
> make and shell got unmaintainable, so adding to that would be unwise.
> Using python should result in a reduction of lines of code, and much
> clearer lines as well.  I'm not sure that was achieved yet.  Comparing
> hardlink_tree and say this snippet
> 
> -     for dir in $(outdir)/web-root/ ; do  \
> -             cd $$dir && \
> -             for a in `find . -name out-www`; do \
> -                     rsync -a  --link-dest $$a/ $$a/ $$a/.. ; \
> -                     rm -rf $$a ; \
> -             done \
> -     done
> 
> (hardlink_tree does much more?).  Having all logic in python is VERY
> NICE, but I would not hesitate to call system ('rsync ..') from python
> if that would dramatically simplify things.

In my newbie mind, I thought that it would not possible to do what we
need for the docs i18n with the shell/make mix, or that it would be very
slow and disk space consuming, so Python was a good solution. After
having read rsync man page, I feel it may be wrong...


-- 
John Mandereau <address@hidden>