[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [fluid-dev] improving musical timekeeping
From: |
address@hidden |
Subject: |
Re: [fluid-dev] improving musical timekeeping |
Date: |
Mon, 10 Feb 2020 01:51:09 +0100 |
User-agent: |
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 |
2020.02.09. 20:35 keltezéssel, Marcus Weseloh írta:
Hi,
first of all: this is a very interesting topic, thank you for bringing
it up!
My pleasure; and I thank you for the moral support.
I do sympathise with your idea and I don't think it is a very unique
use-case at all. I also use MuseScore, but mostly for rehearsing. So I
let MuseScore play some voices of a piece while I play another voice
on top. And I usually choose sounds with a very short attack phase in
MuseScore, so that the attack phases don't mess with the timing. I
have used the OnTime offset shifting of MuseScore 2 from time to time,
but as MuseScore 3 has crippled the user-interface for that feature
(you can only affect a single note at a time), it is now way too much
hassle for me.
Digression: On this scenario (interactive playing on top of scored
music), my idea is that the playback can be performed "on time" via the
features under discussion, while with the interactive play - this is an
area we did not touch on before - a compromise could be achieved (again
using the musical onset markers) by delaying each interactively played
note/sample just enough to make any and all notes played to be delayed
by the same consistent amount, because I have a strong hunch with a
short but consistent delay between keypress and note onset, the
interactively playing musician's brain will be able to compensate for
that delay by pressing keys earlier so that the interactive notes sound
in time with the notes the computer performs from score. The system can
be made aware of the musical onset moments of the interactively played
notes, and notates those moments instead of the moment of the MIDI
keypress. Using this arrangement even interactively recording a new
track will be synchronized to the score without kludges - if the delay
can be kept short enough to fall within the brain's ability where it can
comfortably compensate for it, One option for this is letting the user
choose the maximum delay, and the system truncates attack phases of
interactively played notes down to that limit. It would be a tradeoff
between completeness of the interactively played notes (e.g. for a
performance - in effect reverting to the current behaviour of no attack
phase compensation) on the one end, and completely omitted attack phases
but best interactive playability (which would be a "best effort" full
attack phase compensation for the case when we only learn about the note
at the moment it should be on).
So I agree: it would be great to have a good solution to this problem.
Let me add in advance - instead of pointing this out at each point -,
that in my opinion this compensation logic is essentially already in
place today, with the sample attack phase hardcoded at value zero:
because the synthesizer does already compute the destination moment -
the output audio stream frame (sample) number - where it has to start
playing the sample from the sample's origin (the first sample = sample
offset zero). What would change is merely aligning a new origin (with an
offset that is potentially non zero) point of the sample with the
destination moment. (Granted, my knowledge of the standard is limited
currently.)
Let's assume that SoundFonts would be extended so that they contain
information about the length of the attack phase and the position of
the first "meaningful" sample point, i.e. the sample point that should
be "on beat". Lets ignore the fact that there are MIDI messages that
affect the sample offset. And lets also ignore that the choice of
which sample is heard can also depend on oscillators.
This however can - and indeed, already is being - precalculated, so is
known (assuming by "sample" in the above sentence you mean a time-series
of measured audio values - and not just one of such value of the series;
afaik the English "sample" can mean both in this context.)
And that for a single note-on, many different samples (with different
attack phases) could be started simultaneously.
It would be a meaningful musical discussion to hash out why this needs
to be done at all, and whether in those cases the sound resulting from
the ensemble of those samples played together ends up having a musical
note onset moment instead of the individual samples: if one tries to
reproduce the sound of a preexisting musical instrument, the sound of
that natural instrument when played has a musically meaningful onset -
so by extension, reproducing the sound of that instrument should have
conceptually the same onset, even if for whatever reason the
reproduction of the sound of the instrument ends up getting actually
constructed from multiple "samples" played in parallel - the sound still
represents the same instrument musically, and therefore ought to be
considered the same musically.
(Whether all instrument types have such musically meaningful moments in
their sounds, or not - for the above assume an instrument that does -,
or which ones there are altogether, is a meaningful musical discussion;
and maybe some instruments will not have any such phases; or some
instruments will have some, while other instruments will have others. As
a side note, this is kind of where I think ADSR also originally came
from, trying to capture some musically meaningful and perceptually
pleasant/necessary phases of an instrument's sound.)
Then the synth would have to start playing the sample *before* the
beat, in order to play it on beat. In other words, the sampler would
have to act on a note-on event before this note-on event is actually
due. And in order to decide which sample to play and how much attack
phase time needs to be compensated, it would have to examine all CC
and other messages that could influence sample choice and sample
offsets leading up to that note-on event. And assuming we don't want
to put an artificial limit on how long an attack phase of a sample is
allowed to be, that effectively means that the synth would need to
know and analyse (or even simulate) the complete MIDI stream right
until the last event before starting the playback.
Correct. For rendering
preexisting/non-interactive/non-realtime/score-based music. The notes
would be known ahead of time, so everything could be
precalculated/simulated. Although an artificial limit on how long an
attack phase is allowed to be would not be good, it could be reasonable
to lessen the burden by allowing the software driving the synthesizer to
speficy an upper bound, beyond which attack phases would get truncated.
The synthesizer could maybe provide a suggestion for this to its client
by precalculating some reasonable value for maximum attack lengths, say
over the entire set of samples the music will be played over. The ideal
solution would be to hide it all, and ask for the entire score in
advance and calculate everything perfectly internally. But for cases
where this is inconvenient, a lookahead window could be used, where the
synthesizer is continuously fed events far ahead of current audio output
time so that there is a reasonable assumption that (almost all) of those
events will still be known early enough to the synthesizer to synthesize
them on time (if not, then it's best effort, so some attacks will end up
getting truncated somewhat: if you whip up an instrument with a two
minute attack phase, you better not send a note-on event before two
minutes into the piece; think polyphony and note stealing - another area
where users can expect degradations in output if they ask for
unreasonable things).
That is obviously impossible for live playback, you say that yourself.
Well, on second thought, yes and no - see digression above.
But it also doesn't fit how synthesizers like FluidSynth are used by
MuseScore, Ardour and similar programs. Because as far as I know,
those programs are MIDI sequencers themselves. In other words: *they*
control which MIDI messages to send and - most importantly - when to
send them. They don't pass a complete MIDI file to Fluidsynth to be
played, but rather send MIDI messages in bursts or as a continuous
stream.
I agree, but I don't consider this a conceptual issue, merely an
API/implementation issue. I don't see any conceptual reason why
MuseScore couldn't send the entire score to the synthesizer in advance,
or at least ahead of time with a reasonable look-ahead window. (And we
can open up discussion about making this more efficient by introducing a
caching mechanism so that when an edit is made, not all
instruments/tracks/MIDI channels have to be transferred to - and parsed
and precalculated by - the synthesizer anew.) Maybe it is possible to
edit the score in MuseScore during playback (but then does anyone really
does that)? If it is possible, that could still be made possible by a
sequencer API that allows adding events with timestamps and returns an
identifier to each event, so that the client can later change or remove
events.
And they have very good reasons to do it that way.
So you do consider this a conceptual issue? What would be those good
reasons? I can't think of any for MuseScore - but I am not an expert
user of MuseScore.
So in my opinion, if we wanted to implement a system like you propose,
it would have to be implemented in the MIDI sequencer. In other words:
in MuseScore, Ardour and all the other programs that use MIDI events
to control synthesizers (which also includes FluidSynths internal
sequencer used to play MIDI files).
In light of the above, do you still think this is the case? And if yes,
which parts would end up in the client (MuseScore, Ardour, etc.), and
which parts (if any) would end up in the synthesizer? Would the client
query the synthesizer about note attack lenghths of each and every note,
just to then turn around and advance (bring earlier) the corresponding
note on event before sending it to the synthesizer? Wouldn't that logic
be better done by the synthesizer itself then?
So maybe all that is needed is better sequencer support for shifting
OneTime offsets for notes, tracks and scores.
I don't know what OneTime offsets are, but if you don't have information
on the per-sample (or per-note/per-pitch) attack lengths of each note,
how much do you shift by? Furthermore, if different pitches end up
having different attack lengths (think resampled samples) there is no
single correct value to shift by... so I'm probably missing some part of
your logic?
MuseScore is definitely lacking in that regard, it needs better
user-interfaces for selecting multiple notes and affecting their
OnTime offset. Maybe even support for some database of popular
soundfonts that lists the OnTime offset for each note of each sample.
MuseScore could then read that database and adjust all notes in a
track automatically, if the user decides that is would make musical sense.
This sidecar file to soundfonts I feel to be a quite workaround-like
approach. It could achieve the timing in the short term, yes (but maybe
still not perfectly? - see above). But I could not imagine it as a
long-term solution: it would be specific to MuseScore, the sidecar files
could get separated from the SoundFonts (misplaced, renamed, mixed up,
not found "oh, I have downloaded this soundfont, but cannot find the
sidecar file and now my music is all badly timed, does anyone know where
to download it from?"), how do you know what value to write into the
sidecar file unless you have a soundfont editor to open up the font (or
you are the soundfont author) - at which moment it becomes much more
easier to mark the moment in the soundfont editor and save that with the
soundfont, than having to manually write it into an external file,
taking care not to make a mistake (different note, diferent velocity
layer, etc.), etc.
Cheers
Marcus
- HuBandiT
- [fluid-dev] improving musical timekeeping, address@hidden, 2020/02/07
- Re: [fluid-dev] improving musical timekeeping, address@hidden, 2020/02/07
- Re: [fluid-dev] improving musical timekeeping, Tom M., 2020/02/08
- Re: [fluid-dev] improving musical timekeeping, address@hidden, 2020/02/08
- Re: [fluid-dev] improving musical timekeeping, Tom M., 2020/02/09
- Re: [fluid-dev] improving musical timekeeping, address@hidden, 2020/02/09
- Re: [fluid-dev] improving musical timekeeping, Marcus Weseloh, 2020/02/09
- Re: [fluid-dev] improving musical timekeeping,
address@hidden <=
- Re: [fluid-dev] improving musical timekeeping, Marcus Weseloh, 2020/02/10
Re: [fluid-dev] improving musical timekeeping, Marcus Weseloh, 2020/02/08