Re: Serious feedback and improvement headroom

[Urs Liska]

> ------------------------------------------------------------------------
> *Von:* David Kastrup <address@hidden>
> *Gesendet:* Fri Apr 04 14:24:28 MESZ 2014
> *An:* Urs Liska <address@hidden>
> *CC:* LilyPond Development Team <address@hidden>, "Jan Warchoł"
> <address@hidden>
> *Betreff:* [SPAM] Re: Serious feedback and improvement headroom
>
> Urs Liska <address@hidden> writes:
>
>     The most interesting aspect of the meeting was that Henle's (only)
>     in-house engraver was present too, and this may become a fruitful
>     contact. He is using Amadeus, a Linux (!) program he bought for 20.000
>     Euro in 1988
>
>
> Well, in 1988 there was no Euro.  I presume that you have just converted
> by some factor.

Of course. He said 40.000 Mark.

> But in 1988 there also was no Linux.  Obviously,
> conversion by a factor would not help with that.

Well, I don't know anything about the history of that program, and it's 
hard to find much on the net.
I know Amadeus does run on Linux, and presumably it was developed on Unix.

>     But beyond the basic music items there are significant conceptual
>     differences in the input language. In general LilyPond's language is
>     more verbose than Amadeus'. A few examples:
>
>     treble F 4/4 is the equivalent to
>
>     \clef treble
>     \key f \major
>     \time 4/4
>
>     (,9 .... ),10_a140140h30
>     seems to be an equivalent to a \shape invocation
>
>     Personally I think that LilyPond's approach is a very good compromise
>     between Amadeus' assembler-like appearance and the confusing verbosity
>     of e.g. XML formats. But this is something Henle's engraver would
>     consider a significant problem because he thinks that he needs
>     considerably less time entering the music in Amadeus.
>
>     This becomes particularly important when it comes to tweaking
>     output. He wrote (my translation): "My first look at LilyPond [through
>     my presentation and a follow-up email] shows similarities to Amadeus,
>     but OTOH I have the suspicion that the operation isn't sufficiently
>     mature to allow economic ans fast work. I think, you just have to
>     input too much information to get an optimal result." But he's also
>     arguing at the level of actually entered number of characters.
>
>
> That's something better addressed by input tools rather than the
> language.  Emacs would be a good building block if you are used to
> UNIX-like systems (and if you ported some large application for generic
> UNIX to GNU/Linux, you probably have a bit of exposure here).  It's been
> a long time since I looked at what lyqi, the "LilyPond quick input" mode
> by, uh, Nicolas Sceaux?, did.

Partly yes. When it comes to creating the input text for tweaks I agree 
that editing tools can do a lot to improve things. But when you have the 
strong opinion that typing "f" is an improvement over typing "fis" 
there's not much editing tools can do for you.

>     As a response to my argument that I prefer explicit pitches (a fis is
>     a fis, regardless of needing a sharp or not, while in Amadeus a pitch
>     is only defining a notehead position) he sais: "I don't understand why
>     I should always enter the accidentals explicitly when they are already
>     defined by the key. When I'm seeing the current key in the score
>     display, I do know that in es major the a is actually an as, isn't it?
>
>
> Oh, he's perfectly right about that.  If you are writing a _score_, it
> is reasonable to write down what you see if your goal is to reflect a
> manuscript.  With LilyPond, the focus is on describing music rather than
> programming output, however.  There are advantages to that since I am
> not fixed in the key signature.  I can easily decide at a late stage of
> publishing whether I'll be writing Bach's "Dorische" in the rather
> unmotivated dorian key or whether I just punt and print it in d minor.
>
> I can use the same source for a number of different styles including the
> Urtext.

That's exactly my opinion. Probably he has never prepared a score for a 
different purpose than a single printed edition, so the flexibility of 
differen accidental styles wouldn't matter to him. Anyway, I find the 
idea of the d major chord <d f a> _very_ strange ...
And that's why I wouldn't want to suggest any changes in that area.

>     I have the impression that LilyPond always needs twice or three times
>     the characters for the same content. That's offputting ..."
>
>
> Introduce him to MusiXTeX.

That's not the point. He's (rightly) only interested in the comparison 
to Amadeus.

>     While I still think that explicit pitches are better for a number of
>     reasons this _is_ the way someone thinks who has to produce
>     1.500-2.000 pages of high-quality scores per year. For him each
>     additional character makes a difference.
>
>
> Input tool problem.

I don't think so.
Take the "ho" example, which is shorter than "\stemUp".
You can define a wrapper and write "\ho", which is still longer than 
"ho". The only thing to come to par with the two characters would be a 
keyboard shortcut. But of course it's not a good idea to create 
shortcuts for all of these commands. That would be way over the top.
I think he is right in saying that with LilyPond you need more typing 
than with Amadeus.

But I think that's not the real point. I think it's a tradeoff between 
number of characters to type, clarity and verbosity. And I do think that 
Lily's approach is generally very good here.

> It only becomes a text format problem if the
> verbosity impacts your _reading_ speed or comprehension.  Which is one
> of the reasons many programmers don't like COBOL.  I don't think that
> LilyPond is really too verbose, and you can of course write a lot of
> shortcuts at the price of making it harder to pass scores around.  But
> should be no problem for in-house use.
>
>     I don't suggest any significant changes in our input syntax. But I
>     want to point out that editing efficiency on that level _is_ an issue
>     we should keep taking into account when it comes to professional
>     work.
>
>
> Current LilyPond input tools, editors, and workflows suck.  That is not
> LilyPond's "responsibility" as a command line tool, but of course it is
> something affecting its uptake nevertheless.
>
>     Another aspect that I found very astonishing is compilation speed. Of
>     course Amadeus files have to be compiled before the result is visible,
>     and this can be automatically done upon save. I think Frescobaldi's
>     behaviour with Ctrl-M and by now the autocompile is an equivalent
>     approach here. But the engraver claims that recompilation of a 30 page
>     scores needs 2/10 of a second on an average computer. So this _is_ a
>     fundamental difference, because he _does_ have practically instant
>     WYSIWYG while still benefitting from text input and the compiled
>     approach. I don't have a clue as to how Amadeus achieves this.
>
>
> There is nothing to it, really.  If your account of Amadeus' history is
> correct, it originates at a time when the typical UNIX workstation it
> would be running on would be a 16MHz 68000 CPU.
>
> This is a professional product used by professionals.  So the core
> functionality will have been written in a manner where the professionals
> are not reduced to waiting weeks for a proof.  My guess would be that
> the order of magnitude would have been something like a minute per page.
> With the performance available at that time, we would not have gotten
> our character metrics out of pango in that time, let alone get anything
> done.
>
> Much of the internals of LilyPond are inefficient, and the glue layers
> and data structures of Scheme also come at a cost (and using
> GUILEv1,
> not exactly renowned to be an efficient Scheme implementation, also
> plays into that).  But Scheme is also giving us building blocks for
> solving a lot of stuff without recompilation.

OK, that may well be. But in effect it's a usability issue that 
shouldn't be underestimated. near-realtime WYSIWIG is an important 
factor, particularly when you have to tweak things iteratively by 
trial-and-error. Against Finale users we can always say that the 
compiled approach has inherent advantages that are worth the waiting 
time. But in this case that doesn't count.

>     What comes to mind at first is a very smart approach as to what has to
>     be recalculated and what not. As Amadeus also creates high-quality
>     engraving I can't really imagine that this difference is only due to
>     efficient programming. Although I can imagine that a program developed
>     in the 1980s has a significantly stricter approach to runtime
>     efficiency.
>
>
> We are not talking "stricter approach" here.  We are talking
> "requirement".  One of the advantages of slow iron is that "I'll worry
> about the performance implications of this some other time" is not an
> option.  And that's pervasive in LilyPond.
>
>     But I can also imagine that it has to do with the way C++ and Scheme
>     code interact with LilyPond.
>
>
> Both C++ and Scheme take their toll, yes.
>
>     In another context I see a similar thing with LaTeX: Compiling a file
>     with lualatex and fontspec takes longer by orders of magnitude than
>     with plain latex.
>
>
> If you want a real eye-opener, run weave tex.web && tex tex.tex and take
> a look at the number of seconds taken per page (assuming all fonts are
> available).  That's the performance needed in the 80s.  LaTeX is not
> even close.  It's of course still damn fast when compared with today's
> expectations since it is running on a damn efficient engine, and even if
> this
> engine has to through a lot of ridiculous hoops, it gets the job
> done faster than current programs.
>
>     So maybe we really have a conceptual issue with the efficiency of
>     LilyPond's runtime work.
>
>
> In the current environment?  I don't think all that much.  The important
> thing remains the creative process, and that does not (or should not)
> involve running LilyPond.

I don't understand what you mean. You mean the creative process of 
preparing the content of a score takes more time than compiling the 
LilyPond file?
We're talking about producing an average of ~10 publication quality 
pages per working day, and for this it does make a difference if you 
have to wait 0.2 or 12 seconds to get visual feedback for your editing.

> We _are_ too slow in our backend.  No question about that.  Parsing a
> file is much faster than typesetting it.  And the parsing still could be
> sped up considerably regarding the Scheme/LilyPond synchronization.
>
>     As I can't imagine that we can speed up LilyPond's processing time by
>     95% I really think we should find ways for partial recompilation. This
>     is reportedly also the approach the new Steinberg application
>     takes. For example when entering music I could do with only updating
>     the currently interesting section (maybe a system would be a good unit
>     for that).
>
>
> LilyPond is a command line application.  Partial recompilation does not
> really map sensibly to it.

But there might be steps that can help. I will write more about that in 
response to Jan-Peters message.

> If you write using LaTeX, check out
> <URL:http://www.gnu.org/software/auxtex/preview-latex>.  That's
> technology that had to run on 133MHz CPUs at the height of its
> development and shows interesting and actually usable approaches to
> editor-supported partial compilation.  It should be a reasonably doable
> effort (actually also a nice GSoC-size project) to make preview-latex
> available for LilyPond-book.  In a first step, for LaTeX documents, in a
> second one, working with LilyPond
> documentation in Texinfo.  It would
> make editing our docs so much more pleasant that it isn't funny.
>
> But I digress.  Check out its usability for typical math documents and
> their editing, and in particular recreating formula displays after
> editing.
>
> Most particularly: check out the performance when dvipng is installed
> and you are _not_ going through PDFLaTeX but rather straight LaTeX and
> DVI files.  That's an 80s data flow rather than "modern font handling",
> and it's basically instantaneous.
>
>     Maybe it would be possible to create compilation modes that have a
>     significantly sloppier approach to breaking. There are many stages in
>     the development of a score where I don't care at all about the layout
>     of the complete score. I recall that in Finale you could (probably:
>     can) switch between score and roll mode (however this has been
>     called).
>
>
> You can do that in LilyPond as well.  The resulting output is not
> necessarily all that handy for typical previewers.
>
>     Using strategies with skipTypesetting can help a lot in saving
>     compilation time. But it would be an extremely useful enhancement if
>     we could get some kind of partial recompilation _within_ the context
>     of an existing score.
>
>
> That kind of stuff needs to get integrated into editing tools and modes
> since only editing tools and modes have the effortless information about
> what is currently on-screen and/or edited.
>
>     [I just got another email indicating that Amadeus will by default
>     recalculate the current page only, and this works in near instant
>     time. I think with this we can get to an area where calculating
>     efficiency can make such a difference.]
>
>
> Still needs integration with the editing tool.
>
>     What I would vote for very much is an option to recalculate the
>     current system only, but with outputting the complete score anyway
>     (however it may be determined what "current" is). This should work in
>     many cases when a modification doesn't change line breaking. And it
>     would dramatically increase the editing experience.
>
>
> There is little point in investing a lot of effort here when the current
> editing tools do not even deal with skipTypesetting and other things.

Well, skipTypesetting is not really what would be needed.
I can manually enter the three commands to let LilyPond compile one 
system of an existing score. And we surely can talk Frescobaldi to do 
that automatically. But the result won't be that single system 
re-engraved but the music of the system treated as the beginning of the 
score, with titles and indentation, instrumentnames and whatnot.

> Frescobaldi is the only LilyPond text editor that is actively developed.

And I would try hard to help with that.
Not because I would want to talk that engraver into using LilyPond, but 
because that's only a new push toward an issue we have seen for quite 
some time.

As said I'll write some more in another email.

Urs

> Denemo could probably make use of some of that
> technology, but as it has
> its own display engines, there is not a lot of pressure to get partial
> updates for it.
>
> --
> David Kastrup