Re: [bug-gnulib] Re: speed up gnulib-tool a bit

[Bruno Haible]

Ralf Wildenhues wrote:
> Hmm, ok.  But traditional shells execute this construct
>   while read foo
>   do
>     $whatever
>   done <$file
> 
> in a subshell, too, for example Solaris 10 sh.

Ouch. Looking at the variables, it indeed behaves like a subshell.
Looking at the process id, it does not...!

$ cat script.sh 
echo $$
echo foo > foo
var="initial value"
while read foo
do
  echo foo $$
  var="set by code inside while"
done < foo
echo $$
echo "var = $var"

$ /bin/sh script.sh 
3908
foo 3908
3908
var = initial value

$ /usr/xpg4/bin/sh script.sh
3910
foo 3910
3910
var = set by code inside while

> To be safe, I think you 
> need to `exec <$file', or, restoring stdin, something like this:
>   exec 5<&1 <$file
>   while read r
>   do
>     $whatever
>   done
>   exec <&5 5<&-

Astonishing. I never saw this kind of low-level shell programming.
Applied; see attached patch.

> > Where can I download the shell script profiler that you used?
> 
> Hehe.  Several crude methods, plus some experience with huge shell
> scripts: Either take a slow terminal emulator and watch `sh -x script'
> fly by, or dump its output, and pipe it through
>   sort | uniq -c | sort -k1n
> 
> wher both the short tail and any long, similar regions are interesting.

Thanks for all these tips!

> I think it may be possible to shave off another good part, but the next
> changes probably have a higher source code change to improvement ratio
> (caching func_lookup_file results could help; or rewriting func_get_*
> to work on lists of modules; both seem like not too high hanging fruit).

Uhh, I would be a bit reluctant to apply the second type of changes, since
it changes the program structure for no other reason than speed. The major
bottleneck at this time are the autom4te / m4 invocations after gnulib-tool.
(I'm turning off autom4te's caching in my ~/.autom4te.cfg since temporary
directories would be acceptable for me in /tmp, but not in the middle
of my source code directories.)

Caching func_lookup_file results would be acceptable.

Bruno


*** gnulib-tool 18 Sep 2006 13:07:37 -0000      1.165
--- gnulib-tool 18 Sep 2006 15:13:46 -0000
***************
*** 1672,1688 ****
    LC_ALL=C join -t"$delimiter" -v2 "$tmp"/old-files "$tmp"/new-files \
      | sed -e "$sed_take_last_column" \
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_new_files" > 
"$tmp"/added-files
!   while read g f; do
!     func_add_or_update
!   done < "$tmp"/added-files
    # Then the files that are in new-files and in old-files:
    already_present=true
    LC_ALL=C join -t"$delimiter" "$tmp"/old-files "$tmp"/new-files \
      | sed -e "$sed_take_last_column" \
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_new_files" > 
"$tmp"/kept-files
!   while read g f; do
!     func_add_or_update
!   done < "$tmp"/kept-files
  
    # Command-line invocation printed in a comment in generated gnulib-cache.m4.
    actioncmd="gnulib-tool --import"
--- 1672,1698 ----
    LC_ALL=C join -t"$delimiter" -v2 "$tmp"/old-files "$tmp"/new-files \
      | sed -e "$sed_take_last_column" \
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_new_files" > 
"$tmp"/added-files
!   { # Rearrange file descriptors. Needed because "while ... done < ..."
!     # constructs are executed in a subshell e.g. by Solaris 10 /bin/sh.
!     exec 5<&1 < "$tmp"/added-files
!     while read g f; do
!       func_add_or_update
!     done
!     exec 1<&5 5<&-
!   }
    # Then the files that are in new-files and in old-files:
    already_present=true
    LC_ALL=C join -t"$delimiter" "$tmp"/old-files "$tmp"/new-files \
      | sed -e "$sed_take_last_column" \
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_new_files" > 
"$tmp"/kept-files
!   { # Rearrange file descriptors. Needed because "while ... done < ..."
!     # constructs are executed in a subshell e.g. by Solaris 10 /bin/sh.
!     exec 5<&1 < "$tmp"/kept-files
!     while read g f; do
!       func_add_or_update
!     done
!     exec 1<&5 5<&-
!   }
  
    # Command-line invocation printed in a comment in generated gnulib-cache.m4.
    actioncmd="gnulib-tool --import"
***************
*** 2069,2087 ****
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_files" \
      | LC_ALL=C sort \
      > "$tmp"/files
!   while read g f; do
!     func_lookup_file "$f"
!     if test -n "$lookedup_tmp"; then
!       cp -p "$lookedup_file" "$testdir/$g"
!     else
!       ln "$lookedup_file" "$testdir/$g" 2>/dev/null ||
!       if test -z "$symbolic"; then
          cp -p "$lookedup_file" "$testdir/$g"
        else
!         ln -s "$lookedup_file" "$testdir/$g"
        fi
!     fi
!   done < "$tmp"/files
  
    # Create $sourcebase/Makefile.am.
    mkdir -p "$testdir/$sourcebase"
--- 2079,2102 ----
      | sed -e "s,^.*\$,&$delimiter&," -e "$sed_rewrite_files" \
      | LC_ALL=C sort \
      > "$tmp"/files
!   { # Rearrange file descriptors. Needed because "while ... done < ..."
!     # constructs are executed in a subshell e.g. by Solaris 10 /bin/sh.
!     exec 5<&1 < "$tmp"/files
!     while read g f; do
!       func_lookup_file "$f"
!       if test -n "$lookedup_tmp"; then
          cp -p "$lookedup_file" "$testdir/$g"
        else
!         ln "$lookedup_file" "$testdir/$g" 2>/dev/null ||
!         if test -z "$symbolic"; then
!           cp -p "$lookedup_file" "$testdir/$g"
!         else
!           ln -s "$lookedup_file" "$testdir/$g"
!         fi
        fi
!     done
!     exec 1<&5 5<&-
!   }
  
    # Create $sourcebase/Makefile.am.
    mkdir -p "$testdir/$sourcebase"