[Gcl-devel] Re: (si::sgc-on nil)

[Camm Maguire]

Greetings!  And thank you for your very helpful feedback!

The problem is indeed narrower, at least a conjuction of
multiply-stacks, sgc, and save, from my current understanding.
I.e. you can also leave sgc on and remove the multiply-stacks in the
save.lisp of nqthm-1992 and still be fine.  The default stacks have
been greatly expanded, so you probably don't need this, at least for
now.  I think the situation can be fixed relatively simply to restore
proper functioning even with a combination of these three.  From what
I can tell, this issue has arisen from changes I made for Matt
Kauffman in closing the gbc leak he reported for sgc contiguous
pages.  

For those reading along, the story goes like this -- when one does a
multiply-stacks, the lisp stacks move from the statically allocated
area to contiguous page memory allocated by GCL's memory management
system.  Before Matt's report, GCL never collected these pages when
sgc was on, leading to a bignum leak which we fixed.  So now with sgc
on, only writable contiguous pages pass through the gbc, i.e. any path
from a mark origin through a read-only contiguous page to some object
will not result in that object being marked, as is the desired case.
The problem is, when one saves the system, a final garbage collection
is attempted, and any contiguous pages which survive are marked type
t_other.  My suspicion is that these pages wind up being marked
read-only too long, and hence form a false write-barrier to all
objects addressed from the lisp stack, leading to the freeing of used
objects.  I don't understand yet exactly why this is so, but a fix
should be simple -- exempt pages holding the multiplied stack, and
perhaps all t_other pages, from sgc.

I need to think about this a bit more before deciding what's best --
just wanted to let you know for now that you can lose the
multiply-stacks without any (apparent) penalty.

Take care,

Robert Boyer <address@hidden> writes:

> Thanks so much for the suggestion to turn off sgc!  I really appreciate that.
> It does indeed permit the building of pc-nqthm-1992 with the saved image of
> nqthm-1992.
> 
> However, I wonder if sgc is really or entirely the source of the problem. I
> still suspect the save operation somewhat.  Perhaps there is a bug in the
> save operation that causes it not to save properly images in which sgc is on.
> 
> The message below is one I recently sent you, but I send it again because it
> shows that if (to the best of my knowledge and ability) I do everything
> exactly the same in building nqthm-1992 and then pc-nqthm-1992 except the
> intervening save, then the building (and very extensive execution of
> pc-nqthm-1992) works great.  The command (load "make/save.lisp"), which is
> executed in the script below, does do the (si::sgc-on t), as you know.  The
> very extensive computation that is involved in the execution of the
> pc-nqthm-1992 examples would probably have exposed any corruption, in my
> naive opinion, if the corruption was merely cased by sgc.
> 
> Of course I am lying a bit.  The first thing I do in the script below, namely
> 
>   (defun si::save-system (x) nil)
> 
> which I do to by-pass the save, is enough, given the extremely delicate
> nature of gc bugs, to have completely changed the whole situation.
> 
> Thanks very much,
> 
> Bob
> 
> 
> -------------------------------------------------------------------------------
> 
> 
> >From boyer Mon Nov 17 13:26:28 -0600 2003
> To: address@hidden
> Cc: kaufmann
> Subject: save probably
> 
> 
> Here is a tiny bit more evidence that the problem may be a bug in
> si::save-system.
> 
> In a previous message I reported that if one skips the save operation, then
> the apparent compiler bug that Matt pointed to does not arise.
> Unfortunately, that message's transcript ended with a value stack overflow,
> indicating another possible uncertainty about the nature of the problem.
> However, the value stack overflow is just a normal sort of thing.  I
> conjecture that somewhere between our build of gcl-2.3.8 and our build of
> gcl-2.6.1 there may have been some change to the size of the value stack or
> to what gets pushed onto the value stack by gcl.  If I augment the previously
> sent commands with a simple
> 
>   (SETQ SI::*MULTIPLY-STACKS* 16)
> 
> then Matt's *very* extensive example collection replays without any problem.
> 
> To summarize: given a high enough size for the stacks, no bug arises in the
> standard compilation and loading of nqthm-1992 and the standard compilation
> and loading of pc-nqthm-1992 and the execution of the huge set of
> pc-nqthm-1992 examples, provided no save is done, whereas a bug arises at the
> pc-nqthm compilation stage if before that compilation is begun a save and
> restart from the save is done.
> 
> Here is what I typed, and then below that is the output, including evidence
> of completion of all the pc-nqthm examples.
> 
> -------------------------------------------------------------------------------
> Again, for this example, the directory
> 
>   /u/boyer/nqthm-2nd/nqthm-1992
> contains the unzipped and untarred version of
>   ftp://ftp.cs.utexas.edu/pub/boyer/nqthm/nqthm-1992.tar.gz
> 
> and the directory
>   /u/boyer/pc-nqthm-1992
> contains the unzipped and untarred version of
>   ftp://ftp.cs.utexas.edu/pub/boyer/pc-nqthm/pc-nqthm-1992.tar.gz
> 
> 
> cd /u/boyer/nqthm-2nd/nqthm-1992
> make clean
> set LISP=gcl-2.6.1
> $LISP
> (defun si::save-system (x) nil)
> (load "make/compile.lisp")
> C-C C-Z temporary exit to shell
> echo '"'${LISP}'" "'`pwd`'"' > nqthm-1992-tmp
> fg
> (load "make/save.lisp")
> C-C C-Z temporary exit to shell
> cd /u/boyer/pc-nqthm-1992
> make clean
> make clean-giant-examples
> set CMULISP=cmulisp
> fg
> (SETQ SI::*MULTIPLY-STACKS* 16)
> (si::chdir "/u/boyer/pc-nqthm-1992")
> (load "make/compile.lisp")
> C-C C-Z temporary exit to shell
> echo '"'${CMULISP}'" "'`pwd`'"' > pc-nqthm-1992-tmp
> fg
> (load "make/save.lisp")
> (si::chdir "examples")
> (load "dir.lisp")
> (load "driver.lisp")
> 
> -------------------------------------------------------------------------------
> Here is the ouput of the above input, an entirely successful execution of
> many, many pc-nqthm-examples.
> 
> % cd /u/boyer/nqthm-2nd/nqthm-1992
> % make clean
> rm -f make/compile-success nqthm-1992 TAGS
> rm -f *.o *.c *.h *.data *.sbin *.lbin *.fasl *.sparcf \
>  make/tiny-test.* *saved_nqthm-1992 make/tmp.* *.log
> % set LISP=gcl-2.6.1
> % $LISP
> GCL (GNU Common Lisp)  (2.6.1) Sun Nov 16 04:38:18 CST 2003
> Licensed under GNU Library General Public License
> Dedicated to the memory of W. Schelter
> 
> Use (help) to get some basic information on how to use GCL.
> Loading init.lsp
> Finished loading init.lsp
> 
> >(defun si::save-system (x) nil)
> 
> SYSTEM:SAVE-SYSTEM
> 
> >(load "make/compile.lisp")
> 
> Loading make/compile.lisp
> Loading nqthm.lisp
> Finished loading nqthm.lisp
> Loading sloop.o
> start address -T 0x873d000 Finished loading sloop.o
> Loading basis.o
> start address -T 0x874b000 Finished loading basis.o
> Loading genfact.o
> start address -T 0x875ef60 Finished loading genfact.o
> Loading events.o
> start address -T 0x87a5000 Finished loading events.o
> Loading sloop.o
> start address -T 0x8966000 Finished loading sloop.o
> Loading basis.o
> start address -T 0x87ac000 Finished loading basis.o
> Loading genfact.o
> start address -T 0x873c060 Finished loading genfact.o
> Loading events.o
> start address -T 0x8975ea0 Finished loading events.o
> Loading code-1-a.o
> start address -T 0x8989000 Finished loading code-1-a.o
> Loading code-b-d.o
> start address -T 0x89bc000 Finished loading code-b-d.o
> Loading code-e-m.o
> start address -T 0x89d1000 Finished loading code-e-m.o
> Loading code-n-r.o
> start address -T 0x8a11000 Finished loading code-n-r.o
> Loading code-s-z.o
> start address -T 0x89e5660 Finished loading code-s-z.o
> Loading io.o
> start address -T 0x87b4300 Finished loading io.o
> Loading ppr.o
> start address -T 0x875d4e0 Finished loading ppr.o
> 
> [ 0.0 0.0 0.0 ]
> Finished loading make/compile.lisp
> T
> 
> >
> Suspended
> % echo '"'${LISP}'" "'`pwd`'"' > nqthm-1992-tmp
> % fg
> gcl-2.6.1
> (load "make/save.lisp")
> 
> Loading make/save.lisp
> Loading nqthm.lisp
> Finished loading nqthm.lisp
> Loading sloop.o
> start address -T 0x89999c0 Finished loading sloop.o
> Loading basis.o
> start address -T 0x897e000 Finished loading basis.o
> Loading genfact.o
> start address -T 0x87aaea0 Finished loading genfact.o
> Loading events.o
> start address -T 0x89a5760 Finished loading events.o
> Loading code-1-a.o
> start address -T 0x89f9000 Finished loading code-1-a.o
> Loading code-b-d.o
> start address -T 0x8a5b000 Finished loading code-b-d.o
> Loading code-e-m.o
> start address -T 0x8966000 Finished loading code-e-m.o
> Loading code-n-r.o
> start address -T 0x873bbc0 Finished loading code-n-r.o
> Loading code-s-z.o
> start address -T 0x89d0fe0 Finished loading code-s-z.o
> Loading io.o
> start address -T 0x8992a00 Finished loading io.o
> Loading ppr.o
> start address -T 0x8985fe0 Finished loading ppr.o
> 
> [ 0.0 0.0 0.0 ]
> Loading make/tmp.o
> start address -T 0x87a8f60 Finished loading make/tmp.o
> Finished loading make/save.lisp
> T
> 
> >GCL (GNU Common Lisp)  (2.6.1) Sun Nov 16 04:38:18 CST 2003
> Licensed under GNU Library General Public License
> Dedicated to the memory of W. Schelter
> 
> Use (help) to get some basic information on how to use GCL.
> 
> Nqthm-1992.
> Initialized with (BOOT-STRAP NQTHM) on November 17, 2003  13:02:45.
> 
> 
> >
> Suspended
> % cd /u/boyer/pc-nqthm-1992
> % make clean
> rm -f make/compile-success pc-nqthm-1992 TAGS
> rm -f *.o *.c *.h *.data *.sbin *.lbin *.fasl *.sparcf \
>  make/tiny-test.* *saved_pc-nqthm-1992 make/tmp.* *.log
> % make clean-giant-examples
> cd ./examples ; rm -f */*.lisp */*.lib */*.trans */*.err */*.log \
>   */*.proofs* */*.proved */*.STARTED */tmp.* */*.fail \
>   */*.o */*.fasl */*.lbin */*.sbin */*.sparcf \
>   */*.data */*.c */*.h */[0123456789]*
> % set CMULISP=cmulisp
> % fg
> gcl-2.6.1 (wd: ~/nqthm-2nd/nqthm-1992)
> (SETQ SI::*MULTIPLY-STACKS* 16)
> 
> 16
> 
> >GCL (GNU Common Lisp)  (2.6.1) Sun Nov 16 04:38:18 CST 2003
> Licensed under GNU Library General Public License
> Dedicated to the memory of W. Schelter
> 
> Use (help) to get some basic information on how to use GCL.
> 
> Nqthm-1992.
> Initialized with (BOOT-STRAP NQTHM) on November 17, 2003  13:02:45.
> 
> 
> >(si::chdir "/u/boyer/pc-nqthm-1992")
> 
> "/u/boyer/pc-nqthm-1992"
> 
> >(load "make/compile.lisp")
> 
> Loading make/compile.lisp
> Loading pc-nqthm.lisp
> Loading lisp-test.o
> start address -T 0xde99780 Finished loading lisp-test.o
> Finished loading pc-nqthm.lisp
> Loading proclaim-nqthm.lisp
> Finished loading proclaim-nqthm.lisp
> Loading nqthm-patches.o
> start address -T 0x89bad70 Finished loading nqthm-patches.o
> Loading basis-pc.o
> start address -T 0x87ac000 Finished loading basis-pc.o
> Loading macro-commands-aux.o
> start address -T 0x89bc000 Finished loading macro-commands-aux.o
> Finished loading make/compile.lisp
> T
> 
> >
> Suspended
> (wd now: ~/pc-nqthm-1992)
> % echo '"'${CMULISP}'" "'`pwd`'"' > pc-nqthm-1992-tmp
> % fg
> gcl-2.6.1 (wd: ~/nqthm-2nd/nqthm-1992)
> (load "make/save.lisp")
> 
> Loading make/save.lisp
> Loading pc-nqthm.lisp
> Loading lisp-test.o
> start address -T 0x8a39f20 Finished loading lisp-test.o
> Finished loading pc-nqthm.lisp
> Loading defn-sk.o
> start address -T 0x8989000 Finished loading defn-sk.o
> Loading nqthm-patches.o
> start address -T 0x87b29c0 Finished loading nqthm-patches.o
> Loading basis-pc.o
> start address -T 0x8a3b000 Finished loading basis-pc.o
> Loading help.o
> start address -T 0x87b2c20 Finished loading help.o
> Loading utilities.o
> start address -T 0xe01f000 Finished loading utilities.o
> Loading top.o
> start address -T 0xe05c000 Finished loading top.o
> Loading top-nqthm.o
> start address -T 0xe04d000 Finished loading top-nqthm.o
> Loading macro-commands-aux.o
> start address -T 0xe07a000 Finished loading macro-commands-aux.o
> Loading macro-commands.o
> start address -T 0xe09e000 Finished loading macro-commands.o
> 
> Nqthm-1992 mods:  (PC-NQTHM-1992)
> 
> 
> [ 0.0 0.0 0.0 ]
> Loading make/tmp.o
> start address -T 0x8a4f2e0 Finished loading make/tmp.o
> Finished loading make/save.lisp
> T
> 
> >(si::chdir "examples")
> 
> "examples"
> 
> >(load "dir.lisp")
> 
> Loading dir.lisp
> 
> Loading dir.lisp.
> Evaluating this form: 
> 
> (PROGN
>   (DEFPARAMETER *PC-NQTHM-EXAMPLES-DIR* "./")
>   (DEFPARAMETER *PC-NQTHM-SOURCE-DIR* "../")) 
> Finished loading dir.lisp.
> Finished loading dir.lisp
> T
> 
> >(load "driver.lisp")
> 
> Loading driver.lisp
> 
> Loading driver.lisp.
> We will assume that the Pc-Nqthm-1992 examples directory at this site is: 
> 
>    ./
> 
> We will also assume that the single character / is used to separate
> subdirectory components under this operating system.
> 
> Starting the Pc-Nqthm-1992 examples November 17, 2003  13:03:46.
> Trying subset.events.
> Successfully finished subset.events.
> Trying arith.events.
> Successfully finished arith.events.
> Trying mergesort.events.
> Successfully finished mergesort.events.
> Trying events1.events.
> Successfully finished events1.events.
> Trying transitive-closure.events.
> Successfully finished transitive-closure.events.
> Trying events2.events.
> Successfully finished events2.events.
> Trying pigeon-hole.events.
> Successfully finished pigeon-hole.events.
> Trying hanoi.events.
> Successfully finished hanoi.events.
> Trying ramsey1.events.
> Successfully finished ramsey1.events.
> Trying ramsey2.events.
> Successfully finished ramsey2.events.
> Trying square.events.
> Successfully finished square.events.
> Trying tsquare.events.
> Successfully finished tsquare.events.
> Trying symmetric-difference.events.
> Successfully finished symmetric-difference.events.
> Trying csb.events.
> Successfully finished csb.events.
> Trying finite-state-machine-example.events.
> Successfully finished finite-state-machine-example.events.
> Trying koenig.events.
> Successfully finished koenig.events.
> Trying ramsey.events.
> Successfully finished ramsey.events.
> Trying sets.events.
> Successfully finished sets.events.
> Trying alists.events.
> Successfully finished alists.events.
> Trying terms.events.
> Successfully finished terms.events.
> Trying generalize.events.
> Successfully finished generalize.events.
> Trying c1.events.
> Successfully finished c1.events.
> Trying c2.events.
> Successfully finished c2.events.
> Trying c3.events.
> Successfully finished c3.events.
> Trying c4.events.
> Successfully finished c4.events.
> Trying c5.events.
> Successfully finished c5.events.
> Trying c-signal.events.
> Successfully finished c-signal.events.
> Trying c-prog2.events.
> Successfully finished c-prog2.events.
> Trying c-loop.events.
> Successfully finished c-loop.events.
> Trying c-if.events.
> Successfully finished c-if.events.
> Trying c-begin.events.
> Successfully finished c-begin.events.
> Trying c-predefined1.events.
> Successfully finished c-predefined1.events.
> Trying c-predefined2.events.
> Successfully finished c-predefined2.events.
> Trying c-predefined3.events.
> Successfully finished c-predefined3.events.
> Trying c-predefined4.events.
> Successfully finished c-predefined4.events.
> Trying c-proc-call1.events.
> Successfully finished c-proc-call1.events.
> Trying c-proc-call2.events.
> Successfully finished c-proc-call2.events.
> Trying ca10.events.
> Successfully finished ca10.events.
> Trying r2.events.
> Successfully finished r2.events.
> Trying gf.events.
> Successfully finished gf.events.
> Trying gp.events.
> Successfully finished gp.events.
> Trying ground-resolution.events.
> Successfully finished ground-resolution.events.
> Trying bags.events.
> Successfully finished bags.events.
> Trying naturals.events.start address -T 0x8ea14e0 
> Successfully finished naturals.events.
> Trying interpreter.events.
> Successfully finished interpreter.events.
> Trying dining.events.
> Successfully finished dining.events.
> Trying fifo.events.
> Successfully finished fifo.events.
> Trying me.events.
> Successfully finished me.events.
> Trying min.events.
> Successfully finished min.events.
> Trying Anrd.events.
> Successfully finished Anrd.events.
> Trying bags.events.
> Successfully finished bags.events.
> Trying naturals.events.start address -T 0xe1113c0 
> Successfully finished naturals.events.
> Trying integers.events.start address -T 0xe10d460 start address -T 0xe35f440 
> Successfully finished integers.events.
> Trying knuth-91.events.
> Successfully finished knuth-91.events.
> Trying knuth-91a.events.
> Successfully finished knuth-91a.events.
> 
> All Pc-Nqthm-1992 tests completed successfully.
> 
> real time : 1184.220 secs
> run time  : 1172.130 secs
> 
> Finished loading driver.lisp.
> Finished loading driver.lisp
> T
> 
> >
> 
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah