Re: segfault in grob.cc

[Jan Nieuwenhuizen]

John Galbraith <address@hidden> writes:

> After digging around, it seems that os.system in the python library calls
> system(3) (no big surprise there) which in turns calls /bin/sh to execute
> the command.  On linux, /bin/sh is actually bash. 

Ok, a default setup.

> I appended 'ulimit -a' to the os.system command so I could see what my
> resources were.  It turns out that the stack size is <2K, which is an odd
> number (I don't know where it comes from)  because from the shell it
> reports 8K.

That's strange; I seem to have 8Kb even under os.system ('bash').

> If I also append  'ulimit -s unlimited', then ly2dvi
> *works* all the way through.   It also seems to work with 8192 instead
> of unlimited.  

Ok, that's good news.  A stack of 2 Kb of stack may just be a bit too
small.

> These experiments have been done with about half my score commented
> out.  On the full score, with an unlimited stack, lilypond does not
> technically crash, but does descend into swapping hell after it says
> "paper output to ...".  Would there be something that causes
> exponential growth in memory size (stack or otherwise) as a function
> of input complexity?

Not exponential, `just' linear.  But with large scores, the effect can
be bad.  You just need to get beyond the swapping point, and there's
little hope for LilyPond to finish at all; execution time will
skyrocket.

Try to free as much memory as you can, eg, don't run mozilla, emacs,
X, mysql, apache, X.

You can also play a bit with GUILE settings, but that only helps a
bit.  From lily/main.cc:

  /* Prepare GUILE for heavy memory usage.  If you have plenty memory,
     this may speed up GUILE a bit.  If you're short on memory, these
     settings
    
         export GUILE_INIT_SEGMENT_SIZE_1=36000
         export GUILE_MAX_SEGMENT_SIZE=576000

     may considerably decrease memory footprint (~*0.85), with a small
     execution time penalty (~*1.10).  However, if this 15% gain in memory
     usage prevents swapping, the execution time falls drastically. */

> Is it possible that lilypond actually requires a massive stack for
> large scores, and that incredibly deep recursion in
> Grob::handle_broken_smobs is actually normal behavior?

No, I don't think so; I can't imagine why this would happen.  We
should ask Han-Wen; he'll be back tomorrow or Monday.  LilyPond does
require massive amounts of memory, though.  You could run LilyPond
under proc-time (see http://appel.lilypond.org/software/proc-time.c)
to see how much memory she wants.

>    So, you probably hit the ulimit of your stack; which causes the
>    segfault.  Is it the same under python as directly under bash?
> 
> So no, it is not.  Why is this?

I don't know.  Possibly, there's another bash init script being run.
Maybe python itself imposes the smaller ulimit, but that would be a
bit ugly?

> What is a broken smob, anyway?  Is it possible that my score has some
> diabolical feature that makes a bunch of those?

Not to worry.  All elements are smobs, which is short for Scheme
Object(s).  A `broken smob' is a graphical object (a grob) that spans
a line-break: it is copied and exists before and after the line
break.  For example: slurs, key changes, clef changes.

> What is the largest score that has been processed with lilypond so
> far?  Mine is 830 measures of a score with 12 parts.  Not quite a
> Mahler symphony, but pretty big.

That's indeed pretty big.  The largest score that I have actually seen
typeset, is the Ouverture Coriolan (now in Mutopia).  It has 314
measures and 12 staffs, or 19 parts.  It needed about 110Mb of memory
to process.  How much memory does your machine have?

> My score also has a lot more meter changes, tuplets, ties, and
> articulations than something like the Coriolan that result in a
> higher density of graphic objects.  Am I on the cutting edge of
> lilypond complexity, or are other users out there who *are* actually
> typesetting Mahler or something like that?

You're pretty much on the cutting edge, as far as I know.  Maybe not
of complexity, but certainly with respect to size.  Do you use much
property settings?  It may pay to only use \override and \revert,
memory consumption wise.

We may have to work with you to investigate this deep stack issue
further.

Greetings,
Jan.

-- 
Jan Nieuwenhuizen <address@hidden> | GNU LilyPond - The music typesetter
http://www.xs4all.nl/~jantien       | http://www.lilypond.org