Re: [fluid-dev] FluidSynth backend for SDL_mixer

[David Henningsson]

On 2010-10-06 01:10, James Le Cuirot wrote:
> Hi guys,
>
> First, some good news. I've almost finished implementing a FluidSynth
> backend for SDL_mixer. This will bring long-overdue SoundFont support
> to SDL_mixer as its bundled version of Timidity is very old and doesn't
> support SoundFonts. I just need some help to complete it.

Cool, thanks for the work!

> I'm working with FluidSynth at the sequencer level. Fortunately, some
> of the "native MIDI" code already present in SDL_mixer does the hard
> work of turning the raw MIDI data into a list of event structs.

I assume you had a look at the midi file player and decided it didn't 
work for you?
What version of FluidSynth are you testing with?

> Because the fluid_event_t instances don't get scheduled when they are
> created, I need to store the times somewhere. This struct does have a
> time field but the struct implementation is hidden and the
> fluid_event_get_time and fluid_event_set_time functions are private.
> Wrapping this in another struct would be a PITA so I've cheated and
> declared those private functions to make them available. It would be
> useful if these functions weren't private. I think there are
> potentially other situations where you might want to store the event
> time without actually scheduling the event immediately.

Hmm, because they are private, that means that we are free to change the 
implementation (and even remove the field) between FS versions. I'm not 
sure I want to give up that freedom right away, what does the rest of 
the list think here?

Also, I'm not exactly seeing why wrapping this in another struct would 
be "a PITA", but you can also malloc another array next to the event 
array containing the times if that works better for you. That would also 
stop the overwriting problem you have.

> That isn't the end of the story though because when I do finally
> schedule the event with fluid_sequencer_send_at, the event's time
> property gets overwritten. This is a problem because if the song loops,
> I have to schedule the event again and what was previously a relative
> time has now become an absolute time. I don't think it is necessary to
> overwrite this property because the given event gets copied in the
> _fluid_seq_queue_pre_insert function anyway. With a small adjustment,
> the modification could just be made to the copy instead. For now, I'm
> storing the time before calling fluid_sequencer_send_at and then
> setting it again afterwards.
>
> I haven't yet dealt with SYSEX events because I've never coded around
> MIDI before and this area is a little more complicated so help would be
> appreciated here. However, I've noticed that one of the other MIDI
> backends deals with SYSEX tempo changes and there doesn't seem to be a
> FluidSynth event function for that.
>
> The basic tempo isn't right yet because I haven't got a clue what to
> pass to fluid_sequencer_set_time_scale. I have the PPQN and I've tried
> passing this as well as variations on this but nothing seems to work
> right for both the games I'm testing, D1X-Rebirth and Rise of the Triad.

I think you'll have to look at fluid_player_set_midi_tempo and do 
something similar, as you're doing your own midi file parser, that is, 
combine PPQN with the MIDI_SET_TEMPO meta event.

> In the case of Rise of the Triad, there is an additional problem
> because SDL_mixer is being opened at 11025Hz, which is below what
> FluidSynth supports. As a result, FluidSynth outputs the sound at
> 22050Hz instead and it gets played at half the speed and pitch. I tried
> simply changing the minimum rate in FluidSynth and that worked fine so I
> am politely asking if you could make this change as many games are
> sampled at 11025Hz.

Okay, no problem. I have changed it to 8 kHz. I'm feeling sorry for the 
Rise of the Triad users though, as they will lose all the treble ;-)

> D1X-Rebirth's pitch is also out but it's only slightly lower. If I
> hadn't compared the game's output with standalone FluidSynth and
> Timidity, I wouldn't have noticed. In this case, I'm not sure what the
> problem is. It seems to be outputting at 44100Hz as it should. Any
> ideas?

Sample rate mismatch between 44100 Hz and 48000 Hz, perhaps?

> There is also the issue of sample format. SDL_mixer is supposed to be
> able to output sound in signed/unsigned 8/16-bit litte/big integer
> format. FluidSynth only seems to support signed 16-bit integer and
> float format. The Timidity backend code has some functions to convert
> from signed 32-bit integer to the aforementioned formats but they're
> greek to me and I don't know whether they'd be useful here.
>
> This is complicated by the fact that SDL_mixer is also supposed to
> support multi-channel output. According to the docs, FluidSynth only
> supports this when outputting in float format. It also only outputs as
> stereo pairs but what if SDL_mixer requests mono? Or 5.1? Does 5.1
> count as 2 stereo pairs or 3 and do I have to handle this in some
> special way?

FluidSynth's native format is single- or double-precision floating 
point, depending on configuration. There is also a convenience function 
for converting to 16-bit signed.

I'm quite certain that there are SDL functions for converting to the 
other sample formats you need...?

For the multi-channel stuff, that is not 5.1 - at least not today. 
That's mostly meant for professional audio (think studio), when you want 
separate output for separate MIDI channels.

> Sorry for the barrage of questions but I'm keen to get this going. :)

No problem, feel free to ask. I'm glad to see FluidSynth used in all 
kinds of environments!

(Btw, one thing I spotted: you probably want to free the settings 
instance with delete_fluid_settings instead of just calling "free".)

// David