Re: Serious feedback and improvement headroom

[David Kastrup]

Urs Liska <address@hidden> writes:

> The most interesting aspect of the meeting was that Henle's (only)
> in-house engraver was present too, and this may become a fruitful
> contact. He is using Amadeus, a Linux (!) program he bought for 20.000
> Euro in 1988

Well, in 1988 there was no Euro.  I presume that you have just converted
by some factor.  But in 1988 there also was no Linux.  Obviously,
conversion by a factor would not help with that.

> But beyond the basic music items there are significant conceptual
> differences in the input language. In general LilyPond's language is
> more verbose than Amadeus'. A few examples:
>
> treble F 4/4 is the equivalent to
>
> \clef treble
> \key f \major
> \time 4/4
>
> (,9 .... ),10_a140140h30
> seems to be an equivalent to a \shape invocation
>
> Personally I think that LilyPond's approach is a very good compromise
> between Amadeus' assembler-like appearance and the confusing verbosity
> of e.g. XML formats. But this is something Henle's engraver would
> consider a significant problem because he thinks that he needs
> considerably less time entering the music in Amadeus.
>
> This becomes particularly important when it comes to tweaking
> output. He wrote (my translation): "My first look at LilyPond [through
> my presentation and a follow-up email] shows similarities to Amadeus,
> but OTOH I have the suspicion that the operation isn't sufficiently
> mature to allow economic ans fast work. I think, you just have to
> input too much information to get an optimal result."  But he's also
> arguing at the level of actually entered number of characters.

That's something better addressed by input tools rather than the
language.  Emacs would be a good building block if you are used to
UNIX-like systems (and if you ported some large application for generic
UNIX to GNU/Linux, you probably have a bit of exposure here).  It's been
a long time since I looked at what lyqi, the "LilyPond quick input" mode
by, uh, Nicolas Sceaux?, did.

> As a response to my argument that I prefer explicit pitches (a fis is
> a fis, regardless of needing a sharp or not, while in Amadeus a pitch
> is only defining a notehead position) he sais: "I don't understand why
> I should always enter the accidentals explicitly when they are already
> defined by the key. When I'm seeing the current key in the score
> display, I do know that in es major the a is actually an as, isn't it?

Oh, he's perfectly right about that.  If you are writing a _score_, it
is reasonable to write down what you see if your goal is to reflect a
manuscript.  With LilyPond, the focus is on describing music rather than
programming output, however.  There are advantages to that since I am
not fixed in the key signature.  I can easily decide at a late stage of
publishing whether I'll be writing Bach's "Dorische" in the rather
unmotivated dorian key or whether I just punt and print it in d minor.

I can use the same source for a number of different styles including the
Urtext.

> I have the impression that LilyPond always needs twice or three times
> the characters for the same content. That's offputting ..."

Introduce him to MusiXTeX.

> While I still think that explicit pitches are better for a number of
> reasons this _is_ the way someone thinks who has to produce
> 1.500-2.000 pages of high-quality scores per year. For him each
> additional character makes a difference.

Input tool problem.  It only becomes a text format problem if the
verbosity impacts your _reading_ speed or comprehension.  Which is one
of the reasons many programmers don't like COBOL.  I don't think that
LilyPond is really too verbose, and you can of course write a lot of
shortcuts at the price of making it harder to pass scores around.  But
should be no problem for in-house use.

> I don't suggest any significant changes in our input syntax. But I
> want to point out that editing efficiency on that level _is_ an issue
> we should keep taking into account when it comes to professional
> work.

Current LilyPond input tools, editors, and workflows suck.  That is not
LilyPond's "responsibility" as a command line tool, but of course it is
something affecting its uptake nevertheless.

> Another aspect that I found very astonishing is compilation speed. Of
> course Amadeus files have to be compiled before the result is visible,
> and this can be automatically done upon save. I think Frescobaldi's
> behaviour with Ctrl-M and by now the autocompile is an equivalent
> approach here. But the engraver claims that recompilation of a 30 page
> scores needs 2/10 of a second on an average computer. So this _is_ a
> fundamental difference, because he _does_ have practically instant
> WYSIWYG while still benefitting from text input and the compiled
> approach.  I don't have a clue as to how Amadeus achieves this.

There is nothing to it, really.  If your account of Amadeus' history is
correct, it originates at a time when the typical UNIX workstation it
would be running on would be a 16MHz 68000 CPU.

This is a professional product used by professionals.  So the core
functionality will have been written in a manner where the professionals
are not reduced to waiting weeks for a proof.  My guess would be that
the order of magnitude would have been something like a minute per page.
With the performance available at that time, we would not have gotten
our character metrics out of pango in that time, let alone get anything
done.

Much of the internals of LilyPond are inefficient, and the glue layers
and data structures of Scheme also come at a cost (and using GUILEv1,
not exactly renowned to be an efficient Scheme implementation, also
plays into that).  But Scheme is also giving us building blocks for
solving a lot of stuff without recompilation.

> What comes to mind at first is a very smart approach as to what has to
> be recalculated and what not. As Amadeus also creates high-quality
> engraving I can't really imagine that this difference is only due to
> efficient programming. Although I can imagine that a program developed
> in the 1980s has a significantly stricter approach to runtime
> efficiency.

We are not talking "stricter approach" here.  We are talking
"requirement".  One of the advantages of slow iron is that "I'll worry
about the performance implications of this some other time" is not an
option.  And that's pervasive in LilyPond.

> But I can also imagine that it has to do with the way C++ and Scheme
> code interact with LilyPond.

Both C++ and Scheme take their toll, yes.

> In another context I see a similar thing with LaTeX: Compiling a file
> with lualatex and fontspec takes longer by orders of magnitude than
> with plain latex.

If you want a real eye-opener, run weave tex.web && tex tex.tex and take
a look at the number of seconds taken per page (assuming all fonts are
available).  That's the performance needed in the 80s.  LaTeX is not
even close.  It's of course still damn fast when compared with today's
expectations since it is running on a damn efficient engine, and even if
this engine has to through a lot of ridiculous hoops, it gets the job
done faster than current programs.

> So maybe we really have a conceptual issue with the efficiency of
> LilyPond's runtime work.

In the current environment?  I don't think all that much.  The important
thing remains the creative process, and that does not (or should not)
involve running LilyPond.

We _are_ too slow in our backend.  No question about that.  Parsing a
file is much faster than typesetting it.  And the parsing still could be
sped up considerably regarding the Scheme/LilyPond synchronization.

> As I can't imagine that we can speed up LilyPond's processing time by
> 95% I really think we should find ways for partial recompilation. This
> is reportedly also the approach the new Steinberg application
> takes. For example when entering music I could do with only updating
> the currently interesting section (maybe a system would be a good unit
> for that).

LilyPond is a command line application.  Partial recompilation does not
really map sensibly to it.

If you write using LaTeX, check out
<URL:http://www.gnu.org/software/auxtex/preview-latex>.  That's
technology that had to run on 133MHz CPUs at the height of its
development and shows interesting and actually usable approaches to
editor-supported partial compilation.  It should be a reasonably doable
effort (actually also a nice GSoC-size project) to make preview-latex
available for LilyPond-book.  In a first step, for LaTeX documents, in a
second one, working with LilyPond documentation in Texinfo.  It would
make editing our docs so much more pleasant that it isn't funny.

But I digress.  Check out its usability for typical math documents and
their editing, and in particular recreating formula displays after
editing.

Most particularly: check out the performance when dvipng is installed
and you are _not_ going through PDFLaTeX but rather straight LaTeX and
DVI files.  That's an 80s data flow rather than "modern font handling",
and it's basically instantaneous.

> Maybe it would be possible to create compilation modes that have a
> significantly sloppier approach to breaking. There are many stages in
> the development of a score where I don't care at all about the layout
> of the complete score. I recall that in Finale you could (probably:
> can) switch between score and roll mode (however this has been
> called).

You can do that in LilyPond as well.  The resulting output is not
necessarily all that handy for typical previewers.

> Using strategies with skipTypesetting can help a lot in saving
> compilation time. But it would be an extremely useful enhancement if
> we could get some kind of partial recompilation _within_ the context
> of an existing score.

That kind of stuff needs to get integrated into editing tools and modes
since only editing tools and modes have the effortless information about
what is currently on-screen and/or edited.

> [I just got another email indicating that Amadeus will by default
> recalculate the current page only, and this works in near instant
> time. I think with this we can get to an area where calculating
> efficiency can make such a difference.]

Still needs integration with the editing tool.

> What I would vote for very much is an option to recalculate the
> current system only, but with outputting the complete score anyway
> (however it may be determined what "current" is). This should work in
> many cases when a modification doesn't change line breaking. And it
> would dramatically increase the editing experience.

There is little point in investing a lot of effort here when the current
editing tools do not even deal with skipTypesetting and other things.

Frescobaldi is the only LilyPond text editor that is actively developed.
Denemo could probably make use of some of that technology, but as it has
its own display engines, there is not a lot of pressure to get partial
updates for it.

-- 
David Kastrup