Re: Autocompilation/LilyPond

[Ian Hulin]

Hi David, Mark,
I am still around, I've not had much time for hacking lately as I've
been getting sick again, and the meds tend to sap the higher brain
functions.


On 09/03/12 19:27, David Kastrup wrote:
> Mark H Weaver <address@hidden> writes:
> 
>> David Kastrup <address@hidden> writes:
>> 
>>>> In the long run, I think this is probably your best way
>>>> forward, but admittedly it would require more work to make
>>>> this transition.
>>> 
>>> The main problem is that it requires such a large
>>> reorganisation of the LilyPond sources that the attempts to do
>>> it in that manner tended to consist of outdated nontrivially
>>> rearranged parts before getting through peer review
>>> successfully.  LilyPond is quite a moving target.
>>> 
>>> Ian Hulin has mostly worked on Guilev2 migration in that
>>> manner, and it has caused him to do lots of futile work, a
>>> major cause of frustration for him and of worry for others
>>> because the large reorganisations made the work hard to
>>> verify.
>>> 
>>> So while this might be the "if everything was written from
>>> scratch, it would make most sense to do it this way" approach,
>>> it has proven less than fabulous as a migration strategy [...]
>> 
>> Okay, understood.  The other alternatives are workable, with the
>> build script written in Scheme probably being the most
>> future-proof of the remaining options.
> 
> Sounds somewhat like it.
> 
>> It occurs to me that if the "lilypond module" has not already
>> been set up,
> 
The module is '(lily). It's set up in the Lily initialization code in
C++ thus:

1. main routine calls scm_boot_guile with callback to C++ routine
main_with_guile.

2. main_with_guile
2.1 sets up scheme LOAD_PATH to prepend our local LilyPond root
directory and the /scm subdirectory (this is so the modules declared
in our kit's scheme files (like scm/display-lily.scm get autoloaded
correctly).
2.2 calls ly_c_init_guile to define the '(lily) module in code and
specify ly_init_ly_module as a callback.

2.2.1 ly_init_ly_callback
2.2.1.1 sets up some internal initialization functions,
2.2.1.2 dumps out some trace code, if required, about what it's doing
2.2.1.3 does the equivalent of  (primitive_load_path "lily.scm") from
code.
2.2.2 (now we're back in ly_c_init_guile). scm_c_define_module has
returned a handle to the new module, step 2.2.1.3 has loaded and
evaluated all the stuff in lily.scm - including the load loop with all
the component scheme file held in scm/*.scm with their problematic
interactions.
2.2.3 do equivalent of (use_modules '(lily))
2.3 (now we're back in main_with_guile).  Set up local C++ constructors
2.4 set up fonts stuff
2.5 set up for multi-file compilation
2.5 process command-line options
2.6 set up handling to operate in server (jail) mode
2.7 look up and call the scheme entry-point in the '(lily) module to
actually do the LilyPond compilations - there is no normal exit point
from here, exiting the image is handled by scheme (exit) calls in the
'(lily) code.

So the "lilypond module" *is* normally set up when running with Guile
1.8, providing nothing breaks when lily.scm is loaded.

> I think it will be.
> 
>> then it will be completely empty and not suitable for compiling 
>> anything, so in that case you'll want to do something closer to
>> this:
>> 
>> (define lilypond-module (make-fresh-user-module)) (module-use!
>> lilypond-module (resolve-interface '(guile))) (set-module-name!
>> lilypond-module '(LILYPOND MODULE NAME)) (set-current-module
>> lilypond-module) (for-each (lambda (base-name) (let
>> ((scm-file-name (string-append base-name ".scm")) (go-file-name
>> (string-append base-name ".go"))) (compile-file scm-file-name 
>> #:output-file go-file-name #:env (current-module) #:opts
>> %auto-compilation-options) (load scm-file-name))) 
>> <LIST-OF-BASE-NAMES>)
>> 
>> BTW, if I wanted to take a look to try to get it working myself,
>> what branch of the lilypond git repo should I look at?
> 
> Believe it or not, master.  There is nothing in the public
> repository that would have more progress regarding Guilev2
> migration (it might make sense to ask Ian whether you can pull
> something from him).  master is always kept in a state where it
> compiles, passes regtests, and builds the documentation, and should
> not trail the "staging" branch (where new changes get pushed) for
> more than a day.  Unless "staging" does not pass the tests, in
> which case the automated push does not proceed.
> 
> You can look for open issues regarding Guilev2 migration in 
> <URL:http://code.google.com/p/lilypond/issues/list?can=2&q=guile>
> 
> Various issue carry patches.  I think the one where Ian ran out of
> steam on last had been 
> <URL:http://code.google.com/p/lilypond/issues/detail?id=2026>
> 
Sorry David, this may be a bit of a Red Herring.
<URL:http://code.google.com/p/lilypond/issues/detail?id=1686> was what
I was working on. 2026 involved trying to package the markup subsystem
within the '(lily) module code so we could load it as a module.
Problems are this is
a) subject to the flakiness of the dynamic load order within lily.scm,
*and*
b) as it defines an internal interpreter, the definition order within
the code defining the markup interpreter macros
(scm/markup-macros.scm)is crucial, especially when some of them
actually assume they can forward-reference macro definitions within
the module. This breaks badly when you try to use byte-compiled code
for this file.
c) there are internal design issues with the LilyPond markup code re
validation within the interpreter and its interactions with
declarations in lily.scm.
I decided I needed to park work on 2026 and re-try implementing 1686
using compile-one-file approach.
> Note that the problems occuring here are not all particularly
> related to Guilev2 or its directory layout.  But whenever there was
> non-trivial delay, Ian had to start reorganizing the source tree
> from scratch or try tracking recent changes back into his version.
> So each of the comparatively small problems in migrating meant that
> he had to start over.  And it meant that it was very hard to verify
> his work since there was a large reorganisation each time.
> 
> So while it may all seem like a huge bunch of unnecessary
> stupidity, the "proper" way to do this has, in practice, shown
> itself to result in motivationally devastating effects.  We need a
> migration strategy with more incremental psychological rewards and
> less fear (namely, something that is better verifiable than a large
> reorganisation of the source).
> 
> We mostly have to rely on working with humans.
> 
I had got some way with 1686.  I've got the changes working in our
main.cc (the stuff I walked through above mostly working, and was
trying this strategy.

For Guile 2, lily.scm is re-written as lily-guile2.scm.  The guile 1
code lily.scm is renamed to lily-guile1.scm.
lily-guile2.scm is uses

I added a shim in guile-init.cc (the one containing ly_c_init_guile
and ly_init_ly_module) to load lily-guile2.scm if running Guile 2, and
lily-guile1.scm if running with Guile 1.8.

I also tried compiling lily-guile2.scm  if running Guile2 and if
lily-guile2.go didn't exist before doing calling load-from-path (this
way I could be sure of loading the .go file if at all possible).

This is where I got stuck.  When compiling the new lily-guile2.scm, it
includes a file called music-functions.scm, which autoloads a module
(scm display-lily), which is defined in scm/display-lily.scm, which in
turn uses a scheme hash table.  The hash table is defined in code
using LY_DEFINE (our version of SCM_DEFINE) in our C++ code, and the
current code contains a (use-modules '(lily)) to actually reference
it. The compilation stage currently dies with a scheme
unbound-variable throw.

The next I was going to try before illness intervened was supplying a
new shim lily.scm to do

(cond-expand
        (guile-2
          (load-from-path "lily-guile2.scm"))
        (guile-1
          (load-from-path "lily-guile1.scm")))
So the autoload from music-functions.scm found something to load and
maybe gave me a more sensible message to see if it *really* couldn't
find the internally declared scheme definition.

Guys, thanks for looking at this, hope this info is useful.  Let me
know if you want any of the code I've changed thus far.

Cheers,

Ian Hulin