Re: [Gcl-devel] GCL exists while running smoe benchmarks

[Jeronimo Pellegrini]

Hello!

Thank you a lot!

Reserving code block space inside the program worked fine.

I tried Setting GCL_MEM_MULTIPLE=0.5 when calling gcl, and the problem
(which I had reported in my previous email) still happened. Also tried 
0.25, and it still happened. It worked with 0.1. The weird thing is that
I have 32Gb RAM on this system, and it's mostly unused.


Setting GCL_MEM_MULTIPLE=2 when calling gcl causes a different error:

;; OPTIMIZE levels: Safety=0 (No runtime error checking), Space=0, Speed=3, 
(Debug quality ignored)
;; Finished compiling /tmp/user/1000/gazonk_10795_0.o.
;; Loading #P"/tmp/user/1000/gazonk_10795_0.o"
;; start address for /tmp/user/1000/gazonk_10795_0.o 0x2493ed70
;; Finished loading #P"/tmp/user/1000/gazonk_10795_0.o"
;; Compiling /tmp/user/1000/gazonk_10795_0.lsp.

Unrecoverable error: value stack overflow.
Emergency reset complete

Also, when I was compiling GCL itself, it used up to 20Gb RAM, took
a couple of hours, but managed to finish.

I wonder if there is a simple way to ask the amount of memory in the
system and then allocate code block space.

And I also see that GCL doesn't perform too well in the benchmarks,
because it spends a lot of time doing GC:

RUN-CVECTOR-SET (SET)
real time       :      7.449 secs
run-gbc time    :      7.360 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
RUN-CVECTOR-GET (GET)
real time       :      1.799 secs
run-gbc time    :      1.799 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
TRAVERSE-CVECTOR-SET (tr-SET)
real time       :     12.069 secs
run-gbc time    :     11.399 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
TRAVERSE-CVECTOR-GET (tr-GET)
real time       :     19.460 secs
run-gbc time    :      9.859 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
RUN-HASH-SET (SET)
real time       :      4.699 secs
run-gbc time    :      4.699 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
RUN-HASH-GET (GET)
real time       :      4.650 secs
run-gbc time    :      4.650 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
TRAVERSE-HASH-SET (tr-SET)
real time       :      5.010 secs
run-gbc time    :      5.000 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
TRAVERSE-HASH-GET (tr-GET)
real time       :      4.889 secs
run-gbc time    :      4.880 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
Plain array (SET)
real time       :      2.829 secs
run-gbc time    :      2.779 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs
Plain array (GET)
real time       :      0.000 secs
run-gbc time    :      0.000 secs
child run time  :      0.000 secs
gbc time        :      0.000 secs

I tried using GCL_GC_PAGE_THRESH=0.8, and it seemed to make no
difference. Are there tools to debug and trace GC behavior?

J.



On Sun, Nov 11, 2018 at 01:14:47PM -0500, Camm Maguire wrote:
> Greetings!
> 
> GCL can now make use of all system memory available.  On certain
> machines (e.g. amd64) code is compiled in the 'medium memory' model for
> efficiency (and reliability), which means that all code must be loaded
> within the same 2Gb.  If your heap exceeds this before load, and your
> existing code space is full, you get this message.
> 
> To reserve more space for code, do
> 
> (progn (setq si::*code-block-reserve* (make-array 50000000 :element-type
>                                'character :static t) nil))
> 
> Alternatively, you can limit your heap growth with environment variables
> as explained on this list -- search for GCL_MEM_MULTIPLE.
> 
> Please let me know if problems persist.  Thanks for checking this out!
> 
> Take care,
> 
> Jeronimo Pellegrini <address@hidden> writes:
> 
> > Now -- could that be loop unrolling that exceeded the max size that PROGN 
> > takes in GCL?
> >
> > J.
> >
> > On November 9, 2018 5:23:05 PM GMT-02:00, Jeronimo Pellegrini 
> > <address@hidden> wrote:
> >
> >  Hello,
> >
> > So -- I had used GCL in the past with Spartns 
> > (https://gitlab.com/jpellegrini/spartns)
> > and it worked fine. However, currently this is what happens:
> >
> > - the released version (Debian package, 2.6.12-80) fails
> >   several Spartns unit tests. This is finem because:
> >
> > - the git checkout does pass all tests! However, when running 
> >   the benchmarks, it exits abnormally:
> >
> > $ GCL_ANSI=1 gcl
> > GCL (GNU Common Lisp)  2.7.0 ANSI    Nov  6 2018 23:26:25
> > Source License: LGPL(gcl,gmp,pargcl), GPL(unexec,bfd,xgcl)
> > Binary License:  GPL due to GPL'ed components: (XGCL READLINE UNEXEC)
> > Modifications of this banner must retain notice of a compatible license
> > Dedicated to the memory of W. Schelter
> >
> > Use (help) to get some basic information on how to use GCL.
> > Temporary directory for compiler files set to /tmp/user/1000/
> >
> >  (si::use-fast-links nil) 
> >
> > NIL
> >
> >  (load "do-benchmarks.lisp")
> >
> > ...
> > ... ;; lots of normal output here, from other functions being compiled
> > ...
> > ;; OPTIMIZE levels: Safety=0 (No runtime error checking), Space=0, Speed=3, 
> > (Debug quality ignored)
> > ;; Finished compiling /tmp/user/1000/gazonk_12854_0.o.
> > ;; Loading #P"/tmp/user/1000/gazonk_12854_0.o"
> >
> > Error: 
> > Signalled by PROGN.
> > Condition in PROGN [or a callee]: INTERNAL-SIMPLE-ERROR: The assertion v && 
> > (unsigned long)(v+sz)<MAX_CODE_ADDRESS
> >  on line 1035 of alloc.c in function alloc_code_space
> >  failed: Success
> >
> > Broken at SYSTEM::BREAK-LEVEL.  Type :H for Help.
> >     1  Return to top level. 
> > SPARTNS>> :bt
> >
> > #0   INVOKE-DEBUGGER {loc0=#<conditions::internal-simple-error.0>} [ihs=21]
> > #1   LOAD-FASL {loc0=#P"/tmp/user/1000/gazonk_12854_0.o",loc1=nil} [ihs=18]
> > #2   LOAD-PATHNAME 
> > {loc0=#P"/tmp/user/1000/gazonk_12854_0.o",loc1=nil,loc2=:error,loc3=nil,loc4=nil...}
> >  [ihs=17]
> > #3   LOAD {loc0=#P"/tmp/user/1000/gazonk_12854_0.o"} [ihs=16]
> > #4   COMPILE {loc0=run-cvector-get,loc1=(nil),loc2=nil} [ihs=15]
> > #5   LOAD-STREAM {} [ihs=11]
> > #6   LOAD-PATHNAME 
> > {loc0=#P"benchmark.lisp",loc1=nil,loc2=:error,loc3=nil,loc4=nil,loc5=#<input
> >  str...} [ihs=10]
> > #7   LOAD {loc0="benchmark.lisp"} [ihs=9]
> > #8   LOAD-STREAM {} [ihs=7]
> > #9   LOAD-PATHNAME 
> > {loc0=#P"do-benchmarks.lisp",loc1=nil,loc2=:error,loc3=nil,loc4=nil,loc5=#<input...}
> >  [ihs=6]
> > #10   LOAD {loc0="do-benchmarks.lisp"} [ihs=5]
> >
> > The benchmarks were written in a very lazy state of mind, so I have
> > a big macro that generates lots of functions. But I wasn't expecting this,
> > since the generated functions are actually simple (nested loops for 
> > accessing
> > vectors and hashtables, mostly)...
> >
> > What does the error mean?
> > It's funny that the error happens not during the execution of the benchmark,
> > but when the function (which is quite simple actually) is being compiled!
> >
> > That something called "MAX_CODE_ADDRESS" has been exceeded makes me 
> > worried! :-)
> >
> > What could I possibly be doing to trigger that?
> >
> > (Anyway -- ECL, SBCL, Clisp, ccl seem to compile and run the benchmarks 
> > with no
> >  problem)
> >
> > The easy way to reproduce:
> >
> > $ git clone https://gitlab.com/jpellegrini/spartns.git
> > $ cd spartns
> > $ GCL_ANSI=1 gcl
> >
> >  (si::use-fast-links nil)
> >
> >  (load "do-benchmarks.lisp")
> >
> > Thanks,
> > J.
> > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > Gcl-devel mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/gcl-devel
> 
> -- 
> Camm Maguire                                      address@hidden
> ==========================================================================
> "The earth is but one country, and mankind its citizens."  --  Baha'u'llah