Strange --jobs=n behavior?

[Robert Mecklenburg]

I'm trying to understand the behavior of make --jobs=<n>.  The manual
makes it fairly obvious, but when I run an example it doesn't seem to
match my understanding.  The example consists of a single
non-recursive makefile building an executable divided across
directories like this:

  ./makefile
  app/player/play_mp3.exe
  lib/codec/codec.o
  lib/db/playlist.o  (source generated by bison)
  lib/db/scanner.o   (source generated by flex)
  lib/db/db.o
  lib/ui/ui.o

Each of the object files has a corresponding (generated) dependency
file included by the makefile.

The test runs the same makefile with --jobs=1, then 2, then 3.  The
following listing shows the output of make when --jobs=1, then in the
first three columns the order in which each of the statements is
executed.  Column one for --jobs=1, column two for --jobs=2, and
column three for --jobs=3.

VVVVVVV - order of statement execution for 1, 2, and 3 jobs

1  2  3   make -f ../ch07-separate-binaries/makefile -j <n>

          ** playlist.d **************************************************
1  1  1   bison -y  --defines ../ch07-separate-binaries/lib/db/playlist.y

2  4  6   mv -f y.tab.c lib/db/playlist.c

3  5  7   mv -f y.tab.h lib/db/playlist.h
          
4  8  8   gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -M lib/db/playlist.c | \
          sed 's,\(playlist\.o\) *:,lib/db/\1 lib/db/playlist.d: ,' > 
lib/db/playlist.d.tmp
          
5  11 12  mv -f lib/db/playlist.d.tmp lib/db/playlist.d
          
          ** scanner.d ***************************************************
6  2  3   flex  -t ../ch07-separate-binaries/lib/db/scanner.l > lib/db/scanner.c

7  12 13  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -M lib/db/scanner.c | \
          sed 's,\(scanner\.o\) *:,lib/db/\1 lib/db/scanner.d: ,' > 
lib/db/scanner.d.tmp

8  14 14  mv -f lib/db/scanner.d.tmp lib/db/scanner.d

          ** play_mp3.d **************************************************
9  3  2   gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -M 
../ch07-separate-binaries/app/player/play_mp3.c | \
          sed 's,\(play_mp3\.o\) *:,app/player/\1 app/player/play_mp3.d: ,' > 
app/player/play_mp3.d.tmp

10 7  5   mv -f app/player/play_mp3.d.tmp app/player/play_mp3.d

          ** ui.d ********************************************************
11 10 9   gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -M ../ch07-separate-binaries/lib/ui/ui.c | \
          sed 's,\(ui\.o\) *:,lib/ui/\1 lib/ui/ui.d: ,' > lib/ui/ui.d.tmp

12 13 11  mv -f lib/ui/ui.d.tmp lib/ui/ui.d

          ** codec.d *****************************************************
13 6  4   gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -M 
../ch07-separate-binaries/lib/codec/codec.c | \
          sed 's,\(codec\.o\) *:,lib/codec/\1 lib/codec/codec.d: ,' > 
lib/codec/codec.d.tmp

14 9  10  mv -f lib/codec/codec.d.tmp lib/codec/codec.d

          ** play_mp3.o **************************************************
15 15 15  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -c -o app/player/play_mp3.o 
../ch07-separate-binaries/app/player/play_mp3.c

          ** libcodec.a **************************************************
16 16 16  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -c -o lib/codec/codec.o 
../ch07-separate-binaries/lib/codec/codec.c

17 20 20  ar rv lib/codec/libcodec.a lib/codec/codec.o
          ar: creating lib/codec/libcodec.a
          a - lib/codec/codec.o

          ** libdb.a *****************************************************
18 17 17  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -c -o lib/db/playlist.o lib/db/playlist.c

19 18 18  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -c -o lib/db/scanner.o lib/db/scanner.c
          ../ch07-separate-binaries/lib/db/scanner.l: In function `yylex':
          ../ch07-separate-binaries/lib/db/scanner.l:9: warning: return makes 
integer from pointer without a cast

20 21 22  ar rv lib/db/libdb.a lib/db/playlist.o lib/db/scanner.o
          ar: creating lib/db/libdb.a
          a - lib/db/playlist.o
          a - lib/db/scanner.o

          ** libui.a *****************************************************
21 19 19  gcc  -I lib -I ../ch07-separate-binaries/lib -I 
../ch07-separate-binaries/include  -c -o lib/ui/ui.o 
../ch07-separate-binaries/lib/ui/ui.c

22 22 21  ar rv lib/ui/libui.a lib/ui/ui.o
          ar: creating lib/ui/libui.a
          a - lib/ui/ui.o

          ** play_mp3 ****************************************************
23 23 23  gcc   app/player/play_mp3.o lib/codec/libcodec.a lib/db/libdb.a 
lib/ui/libui.a   -o app/player/play_mp3


(The lines of asterisks identify "top-level" targets that, I believe,
make will spawn new sub-jobs for.)

First make must build the dependency files, which forms a
synchronization point (make must wait for all jobs to finish so it can
re-read the makefile).

It appears from the numbers, that when --jobs=2 make starts tasks to
build (1) playlist.d, (2) scanner.d, and (3) paly_mp3.d.  When
--jobs=2, it appears make starts tasks to build (1) playlist.d, (2)
play_mp3.d, (3) scanner.d, and (4) codec.d.

In other words, a simple interpretation of the logs indicates that
make is spawning n+1 jobs when --jobs=n.  Paul, any idea what's going
on?

Thanks,
Robert Mecklenburg