Re: [gnulib-tool-py] func_filter_filelist, func_get_automake_snippet_unconditional, etc.

[Bruno Haible]

Hi Dmitriy,

> I've already implemented func_filter_filelist, but I'm not sure that it
> works as follows, so I ask you to describe what this function does.

OK. The documentation is pretty unambiguous, I thought:
# func_filter_filelist outputvar separator filelist prefix suffix 
removed_prefix removed_suffix [added_prefix [added_suffix]]
# stores in outputvar the filtered and processed filelist. Filtering: Only the
# elements starting with prefix and ending with suffix are considered.
# Processing: removed_prefix and removed_suffix are removed from each element,
# added_prefix and added_suffix are added to each element.
# prefix, suffix should not contain shell-special characters.
# removed_prefix, removed_suffix should not contain the characters "$`\{}[]^|.
# added_prefix, added_suffix should not contain the characters \|&.

In Python, outputvar will just be a normal return value.
The $separator is used to delimit the values in outputvar, not used in Python.
$filelist is a list of file names.
Those elements of $filelist that start with the given $prefix and end in the
given $suffix are considered; all others are discarded.
For each considered element: If it starts with $removed_prefix, this
prefix $removed_prefix is removed. If it ends in $removed_suffix, this
suffix $removed_suffix is removed. Then, $added_prefix is added at the
beginning, and $added_suffix is added at the end.

> It is very hard to understand what func_get_automake_snippet_unconditional
> does, so I ask you to describe what it does.

Yeah, the 'sed' script sed_combine_lines is not easy to understand, I guess...

Here it goes. For every module, there is a piece of Makefile.am code
(see the 'Automake' documentation for the syntax), defined in the module
description. To observe its effects, do
  $ ./gnulib-tool --create-testdir --dir=/tmp/testdir module
and inspect the file lib/Makefile.am for a normal module and the file
tests/Makefile.am for a tests module.

When conditional dependencies are enabled, this snippet can be enclosed
in an 'Automake conditional' (a 'if'...'endif' block), but some parts
have to stay outside the 'if'...'endif' block, for reasons that lie in
the way Automake works. func_get_automake_snippet_unconditional returns
this latter code.

Decide yourself whether the return type should be a string with newlines,
or a list of lines.

* If the module argument is a test module (ends in "-tests")
  we append one line of the form
    EXTRA_DIST += some-extra-files
  where these extra files are those files from the file list that start
  with tests/. Example: modules/strncat-tests contains the file list

    tests/test-strncat.c
    tests/unistr/test-strncat.h
    tests/zerosize-ptr.h
    tests/signature.h
    tests/macros.h
    m4/mmap-anon.m4

  Take those that start with tests/ and remove this prefix. You get

    test-strncat.c
    unistr/test-strncat.h
    zerosize-ptr.h
    signature.h
    macros.h

  Now create a single string, with a space as separator. This gives the line:

    EXTRA_DIST += test-strncat.c unistr/test-strncat.h zerosize-ptr.h 
signature.h macros.h

  After this line, we also emit a blank line.

  Evidently, when the list of extra files is empty, we don't need the
  EXTRA_DIST augmentation line nor the blank line; they would be useless.

* For a non-tests module, we
  1. take the Makefile.am snippet (this is the statement in braces, identical
     to func_get_automake_snippet_conditional),
  2. combine lines. You have to know that backslash-newline in Makefile.am
     syntax is equivalent to a single space. For example
       foo += file1\
       file2 \
         file3
       bar =\
         dummy
     becomes
       foo += file1 file2   file3
       bar =   dummy
  3. Extract lines that look like a 'lib_SOURCES' augmentation. That is,
     a line that starts with 'lib_SOURCES', then 0 or more spaces or tabs,
     then a '+=' sign, then again 0 or more spaces or tabs. Then we take the
     rest of such a line.
  4. Then we eliminate comments (that start with '#').
  This gives the list of "already mentioned files".
  We emit an EXTRA_DIST augmentation (again, with space separated elements
  and a blank line afterwards) for those files that are listed in the file
  list, starting with 'lib/', and not among the "already mentioned files".

  For example, modules/read-file contains the Makefile.am snippet

  lib_SOURCES += read-file.c

  So, the "already mentioned files" list is ["read-file.c"]. On the other
  hand, it has the file list

    lib/read-file.h
    lib/read-file.c
    m4/read-file.m4

  Among these, consider only those that start with 'lib/' and remove this
  prefix. Gives

    read-file.h
    read-file.c

  Now, from this list, remove the "already mentioned files". Those that
  remain are: ["read-file.h"]. Now construct an EXTRA_DIST augmentation
  for this set:

    EXTRA_DIST += read-file.h

  followed by a blank line.

> func_determine_use_libtests is a very easy function to understand, so I
> think there are no problems.

Yes, this one is easy: Iterate over the "tests related modules", and
for each one look at the file list. If there is a file name that starts
with 'lib/' we need a libtests.

> I need to ask about license_incompatibilities. As I've understood there is
> a table of incompatible licenses, but I'm not sure what we need here.

If the option --lgpl or --lgpl=2 or --lgpl=3 was given (or recovered from
the gnulib.cache file) [note: no checks to do when the user wants a
GPLed result], then we need to check that the "main modules" [note:
the "tests related modules" are ignored here, as their code ends up in
libtests.a with does not get installed] have a compatible license.
If --lgpl or --lgpl=3, the compatible licenses are:
  'GPLed build tool'
  'public domain'
  'unlimited'
  'unmodifiable license text'
  LGPL
  LGPLv2+
  LGPLv3+
If --lgpl=2, the compatible licenses are only:
  'GPLed build tool'
  'public domain'
  'unlimited'
  'unmodifiable license text'
  LGPLv2+
If some incompatible licenses are found, gnulib-tool reports them all
(not just the first one), and also reports the module. All this nicely
formatted in a table that looks like this:

gnulib-tool: *** incompatible license on modules:
                 xalloc                                            GPL
                 foobarbaz                                         LGPLv3+

The first 17 columns are indentation; the next 50 columns are reserved for
the module name. Then comes the license. If the module name has more than
50 characters, make sure there is still a space before the license:

                 very-long-module-name-that-is-longer-than-anyone-expects GPL
                 foobarbaz                                         LGPLv3+

> I don't want to disturb you with
> a lot of questions, but these questions really help me to do the work more
> quickly.

OK, I explained the coarse functioning. But if there is a detail that is not
clear, don't hesitate to peek in the shell script code.

Bruno