Discussion:
[fluid-dev] API design: fluid_synth_process()
Tom M.
2018-04-26 12:42:17 UTC
Permalink
I thought to release a beta in the nearer future. But I would like to have the API ready before I do so. One function that still strikes me is fluid_synth_process() [1]. It's lacking a real implementation for 15 years. If we dont get it right now, we'll never do, so we better discuss it.

This function is used for pulling synthesized audio from the synth. It's said to become the "generic interface in future versions". Although its usage is discouraged as being still "experimental", it's already established due to being required for the callback function of new_fluid_audio_driver2() [2].

Despite the fact that this function ignores two of its arguments, I just dont know how to properly tell it which channels to map to which output buffers. I'm thinking here in the scope of a "future release" that might support surround audio and rendering effects of each midi channel to separate stereo / surround buffers.

Does anybody have an idea how to deal with this function (preferably without changing its signature)? Any idea of how those two ignored arguments could be used? Would welcome any thoughts (esp. from a user perspective).

Tom

[1]: http://www.fluidsynth.org/api/synth_8h.html#a1ac90e2732aa652679305f78cbd66670
[2]: http://www.fluidsynth.org/api/audio_8h.html#a3959d8add1dea97e507a5ea2c802c0bc
Ceresa Jean-Jacques
2018-05-01 12:49:55 UTC
Permalink
Hello Tom

 
Post by Tom M.
I thought to release a beta in the nearer future. But I would like to have the API ready before I do so.
 

I have noticed that 24 bits support introduces not negligible lost of performance that could be solved relatively easily .

The way to solve this need some minor pre-cleanup and could be also (though not necessarly) slightly related to interpolation method API cleanup possibilities.

I will continue this discussion sooner on a GitHub issue named "24 bit support performance enhancement".

Perhaps it worth to solve this before any beta realease ?

 
Post by Tom M.
One function that still strikes me is fluid_synth_process()..
Sorry, i never used this function, but i will take a look.

jjc.
Post by Tom M.
Message du 26/04/18 14:42
De : "Tom M."
Objet : [fluid-dev] API design: fluid_synth_process()
I thought to release a beta in the nearer future. But I would like to have the API ready before I do so. One function that still strikes me is fluid_synth_process() [1]. It's lacking a real implementation for 15 years. If we dont get it right now, we'll never do, so we better discuss it.
This function is used for pulling synthesized audio from the synth. It's said to become the "generic interface in future versions". Although its usage is discouraged as being still "experimental", it's already established due to being required for the callback function of new_fluid_audio_driver2() [2].
Despite the fact that this function ignores two of its arguments, I just dont know how to properly tell it which channels to map to which output buffers. I'm thinking here in the scope of a "future release" that might support surround audio and rendering effects of each midi channel to separate stereo / surround buffers.
Does anybody have an idea how to deal with this function (preferably without changing its signature)? Any idea of how those two ignored arguments could be used? Would welcome any thoughts (esp. from a user perspective).
Tom
[1]: http://www.fluidsynth.org/api/synth_8h.html#a1ac90e2732aa652679305f78cbd66670
[2]: http://www.fluidsynth.org/api/audio_8h.html#a3959d8add1dea97e507a5ea2c802c0bc
_______________________________________________
fluid-dev mailing list
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Tom M.
2018-05-01 17:57:23 UTC
Permalink
Post by Ceresa Jean-Jacques
I have noticed that 24 bits support introduces not negligible lost of performance that could be solved relatively easily .
Thanks for the hint. Looking forward to filing an issue or PR.
Post by Ceresa Jean-Jacques
Perhaps it worth to solve this before any beta realease ?
Sorry, I do not think it's worth to delay the beta for those
performance issue(s). We'll have plenty of time looking into these
internal issues afterwards. The API is more important for now. Note
however that there's neither a due date for a beta, so if we happen to
have all those refactorings and cleanups ready, we'll have it for the
beta.

Tom
Ceresa Jean-Jacques
2018-05-02 01:29:00 UTC
Permalink
Hi,

 
Post by Tom M.
Despite the fact that this function ignores two of its arguments, I just dont know how to properly tell it which channels to map to which output buffers.
 

fluid_synth_process(int len, int nout, float** out) behaves like fluid_synth_nwrite_float(int len, float** left, float** right). Both functions are intended

to return synthesized buffers for a number of stereo audio channels (synth->audio_channels). fluid_synth_process() seems adapted to send audio to audio card with multiples

stereo audio channels (unlike fluid_synth_write_float(), fluid_synth_write_s16() which are only useful for only one stereo audio channel).

The internal mixer stereo audio channels buffers (from 0 to audio_channels-1) are outputed in the out array in an interleaving format:

(l_audio_chan_0, r_audio_chan_0, l_audio_chan_1, r_audio_chan_1,....). Consequently nout and out array must be set to sufficient size (nout >= 2 x synth->audio_channels).

 

The way a MIDI channel number is mapped to an audio channel number is done inside the fluidsynth mixer in fluid_rvoice_buffers_mix().

During fluid_rvoice_buffers_mix(), the function is instructed by the voice to select (mapping) the audio channel buffer number m to mix this voice.

In the actual version the mapping m is function of the MIDI channel of the voice:

     m= 2 x ( MIDI channel number % buf_count) with

     with buf_count = max (audio_channels, audio_groups).

 

Somes pictures are attached to illustrates this.

Hope the author of this API could tell more about the intended unused parameters int nin, float** in.

jjc
Post by Tom M.
Message du 26/04/18 14:42
De : "Tom M."
Objet : [fluid-dev] API design: fluid_synth_process()
I thought to release a beta in the nearer future. But I would like to have the API ready before I do so. One function that still strikes me is fluid_synth_process() [1]. It's lacking a real implementation for 15 years. If we dont get it right now, we'll never do, so we better discuss it.
This function is used for pulling synthesized audio from the synth. It's said to become the "generic interface in future versions". Although its usage is discouraged as being still "experimental", it's already established due to being required for the callback function of new_fluid_audio_driver2() [2].
Despite the fact that this function ignores two of its arguments, I just dont know how to properly tell it which channels to map to which output buffers. I'm thinking here in the scope of a "future release" that might support surround audio and rendering effects of each midi channel to separate stereo / surround buffers.
Does anybody have an idea how to deal with this function (preferably without changing its signature)? Any idea of how those two ignored arguments could be used? Would welcome any thoughts (esp. from a user perspective).
Tom
[1]: http://www.fluidsynth.org/api/synth_8h.html#a1ac90e2732aa652679305f78cbd66670
[2]: http://www.fluidsynth.org/api/audio_8h.html#a3959d8add1dea97e507a5ea2c802c0bc
_______________________________________________
fluid-dev mailing list
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Tom M.
2018-05-02 14:40:27 UTC
Permalink
fluid_synth_process() behaves like fluid_synth_nwrite_float()
Except for the fact that it doesnt handle fx channels.
Consequently nout and out array must be set to sufficient size (nout >= 2 x synth->audio_channels).
Not necessarily. I think fluid_synth_process() should also be usable for simple stereo mixing, i.e. nout == 2. In fact I believe that no matter how many output buffers the user calls it with, fluid_synth_process should always render all playing voices to those buffers (provided that nout >= 2).

The channel layout indeed can be used by fluidsynths internal rendering engine. I'd suggest to use / extend the current calling convention. As you already said "out" currently contains an array of planar buffers for normal, dry, stereo audio (alternating left and right). Like:

out[0] = left_buffer_channel_1
out[1] = right_buffer_channel_1
out[2] = left_buffer_channel_2
out[3] = right_buffer_channel_2
...
out[ i*2 + 0 ] = left_buffer_channel_i
out[ i*2 + 1 ] = right_buffer_channel_i

where 0 <= i < fluid_synth_count_audio_channels()

Just following to those "normal" audio buffers we could place the effects buffers like:

out [ fluid_synth_count_audio_channels() + 0 ] = left_buffer_fxchannel_1 (currently hardcoded to reverb)
out [ fluid_synth_count_audio_channels() + 1 ] = right_buffer_fxchannel_1 (currently reverb)
out [ fluid_synth_count_audio_channels() + 2 ] = left_buffer_fxchannel_2 (currently chorus)
out [ fluid_synth_count_audio_channels() + 3 ] = right_buffer_fxchannel_2 (currently chorus)

out [ fluid_synth_count_audio_channels() + k*2 + 0 ] = left_buffer_fxchannel_k (if this will ever be added)
out [ fluid_synth_count_audio_channels() + k*2 + 1 ] = right_buffer_fxchannel_k

where 0 <= k < fluid_synth_count_effects_channels()

If surround audio would ever be implemented, channel layout could look like:

out[ i*5 + 0 ] = left_front_buffer_channel_i
out[ i*5 + 1 ] = right_front_buffer_channel_i
out[ i*5 + 2 ] = center_front_buffer_channel_i
out[ i*5 + 3 ] = left_rear_buffer_channel_i
out[ i*5 + 4 ] = right_rear_buffer_channel_i

We could directly use this channel layout for rvoice_mixer internally, provided that the requested number of audio frames to synthesize is a multiple of fluid_synth_get_internal_bufsize() so we dont have to hold any temporary audio buffer.

Disadvantage: Not very user friendly. There is no clear separation between dry and effects channels anymore, like done in fluid_synth_nwrite_float(). This will lead to ambiguous situations: if the user calls fluid_synth_process() with 4 output buffers, is he requesting the first two normal stereo audio channels, or one stereo channel and one effects channel? We could (ab)use the "in" parameter for effect buffers to avoid this ambiguity, unless someone sees a better use-case for the "in" param?

Tom
Ceresa Jean-Jacques
2018-05-02 17:41:12 UTC
Permalink
Post by Tom M.
In fact I believe that no matter how many output buffers the user calls it with, fluid_synth_process should always render all playing voices to those buffers (provided that nout >= 2).
yes, this is what fluid_synth_process() actually does (apart the internal mapping "MIDI channel to ouput buffers").

 
Post by Tom M.
Just following to those "normal" audio buffers we could place the effects buffers
Yes it could be done as these buffers effect availability is also important.

 
Post by Tom M.
out[ i*5 + 0 ] = left_front_buffer_channel_i
out[ i*5 + 1 ] = right_front_buffer_channel_i
out[ i*5 + 2 ] = center_front_buffer_channel_i
out[ i*5 + 3 ] = left_rear_buffer_channel_i
out[ i*5 + 4 ] = right_rear_buffer_channel_i
 

You mean 5.1 (i.e 6 channels)

Sorry i dont' understand this index 5 ?.Does Surround could follow any others know buffers ?

 
Post by Tom M.
We could directly use this channel layout for rvoice_mixer internally, provided that the requested number of audio frames to synthesize is a multiple of fluid_synth_get_internal_bufsize() so we dont >have to hold any temporary audio buffer.
Yes but we need to keep in mind that the requested number of audio frames and the format is often imposed by the host API driver.

Actually the actual mixer internal temporary audio buffer is pretty flexible/performant .

 
Post by Tom M.
Disadvantage: Not very user friendly. There is no clear separation...
Yes, but this shouldn't be an issue as far there is no unknow. This is a matter of documentation .

 
Post by Tom M.
We could (ab)use the "in" parameter for effect buffers to avoid this ambiguity..
Mapping of dry and effect to output buffers is not easy to solve. I don't think that fluidsynth should impose his strategie.

-1) For exemple when the application want to use only one stereo audio channel, it is normal that dry and effect are mixed in the

same output buffers (as fluidsynth does actually).

-2) The oposite exemple of 1 comes when multiples audio channels are availlable and the application want to decide if some effect

     should be mixed with some dry in the same output buffer.

 For now i need time to think how to solve (2) clearly using "in" parameter (or others way).

-Note) Also, i think that in the future, the actual internal "MIDI channel to output buffer" hard coded mapping should be replaced by an API.

 

jjc

 

 

 

 
Post by Tom M.
Message du 02/05/18 16:40
De : "Tom M."
A : "Ceresa Jean-Jacques"
Copie à : "FluidSynth mailing list"
Objet : Re: [fluid-dev] API design: fluid_synth_process()
fluid_synth_process() behaves like fluid_synth_nwrite_float()
Except for the fact that it doesnt handle fx channels.
Consequently nout and out array must be set to sufficient size (nout >= 2 x synth->audio_channels).
Not necessarily. I think fluid_synth_process() should also be usable for simple stereo mixing, i.e. nout == 2. In fact I believe that no matter how many output buffers the user calls it with, fluid_synth_process should always render all playing voices to those buffers (provided that nout >= 2).
out[0] = left_buffer_channel_1
out[1] = right_buffer_channel_1
out[2] = left_buffer_channel_2
out[3] = right_buffer_channel_2
...
out[ i*2 + 0 ] = left_buffer_channel_i
out[ i*2 + 1 ] = right_buffer_channel_i
where 0 <= i < fluid_synth_count_audio_channels()
out [ fluid_synth_count_audio_channels() + 0 ] = left_buffer_fxchannel_1 (currently hardcoded to reverb)
out [ fluid_synth_count_audio_channels() + 1 ] = right_buffer_fxchannel_1 (currently reverb)
out [ fluid_synth_count_audio_channels() + 2 ] = left_buffer_fxchannel_2 (currently chorus)
out [ fluid_synth_count_audio_channels() + 3 ] = right_buffer_fxchannel_2 (currently chorus)
out [ fluid_synth_count_audio_channels() + k*2 + 0 ] = left_buffer_fxchannel_k (if this will ever be added)
out [ fluid_synth_count_audio_channels() + k*2 + 1 ] = right_buffer_fxchannel_k
where 0 <= k < fluid_synth_count_effects_channels()
out[ i*5 + 0 ] = left_front_buffer_channel_i
out[ i*5 + 1 ] = right_front_buffer_channel_i
out[ i*5 + 2 ] = center_front_buffer_channel_i
out[ i*5 + 3 ] = left_rear_buffer_channel_i
out[ i*5 + 4 ] = right_rear_buffer_channel_i
We could directly use this channel layout for rvoice_mixer internally, provided that the requested number of audio frames to synthesize is a multiple of fluid_synth_get_internal_bufsize() so we dont have to hold any temporary audio buffer.
Disadvantage: Not very user friendly. There is no clear separation between dry and effects channels anymore, like done in fluid_synth_nwrite_float(). This will lead to ambiguous situations: if the user calls fluid_synth_process() with 4 output buffers, is he requesting the first two normal stereo audio channels, or one stereo channel and one effects channel? We could (ab)use the "in" parameter for effect buffers to avoid this ambiguity, unless someone sees a better use-case for the "in" param?
Tom
Tom M.
2018-05-02 19:45:54 UTC
Permalink
Post by Ceresa Jean-Jacques
You mean 5.1 (i.e 6 channels)
Sorry i dont' understand this index 5 ?.Does Surround could follow any others know buffers ?
I think each audio channel should receive the full spectrum. fluidsynth shouldnt be in charge of rendering the subwoofer channel. But these are implementation details for surround audio, which is way beyound the scope of this talk. Just wanted to point out the channel layout for fluid_synth_process() if the user wishes to instantiate a surround capable synth.
Post by Ceresa Jean-Jacques
Mapping of dry and effect to output buffers is not easy to solve. I don't think that fluidsynth should impose his strategie.
One might allow buffers to alias. Suppose

"synth.audio-channels" is 1
"synth.effects-channels" is 2

The user would call fluid_synth_process(synth, 64, 0, NULL, 6 out) where out:

out[0] = out[2] = out[4] = left_buf;
out[1] = out[3] = out[5] = right_buf;

and left_buf and right_buf are zeroed buffers to which fluidsynth would simply add the synthesized audio.

If the user only passes 2 buffers, the channels would wrap around and the result would be the same:

out[0] = left_buf;
out[1] = right_buf;
fluid_synth_process(synth, 64, 0, NULL, 2 out)
Post by Ceresa Jean-Jacques
Note) Also, i think that in the future, the actual internal "MIDI channel to output buffer" hard coded mapping should be replaced by an API.
Shouldnt be the user in charge of properly mapping any multichannel rendered output?


Tom
Ceresa Jean-Jacques
2018-05-02 22:26:07 UTC
Permalink
jj >Note) Also, i think that in the future, the actual internal "MIDI channel to output buffer" hard coded mapping should be replaced by an API.
Tom>Shouldnt be the user in charge of properly mapping any multichannel rendered output?
 

I prefer thinking of "MIDI channels mapping" (done in rvoice_buffers_mix()) a different subject that mapping any "multichannel rendered (i.e dry, effects)" to final output that should be done in fluid_synth_process().

 

About MIDI Channel mapping subject: assume "synth.audio-channels" is 2, actually rvoice_buffers_mix() will do this (using hard coded formula, see picture fluid_rvoice_buffers_mix_mapping_1.jpg) :

MIDI channels 0,2,4,6... are mixed  to internal audio channel 0.

MIDI channels 1,3,5,7... are mixed to internal audio channel 1.

Actually this mixing is imposed by the internal mixer hard coded mapping . If the Application wanted (for example) to get MIDI channel 2 mixed to audio channel 1, this is not possible.

The only easy way i see for an application to choose which instrument will be mixed in which speakers is a mapping API (to replace the actual hard coded MIDI channels mapping

done in rvoice_buffers_mix()).

I am aware this is also (like surround) out the scope of the actual fluid_synth_process() talk which relates only about "dry" and "effects"  mapping to final ouptut channels. Simply the need of "easy  mapping (as possible)"  is common to both subject. I hope that any reader will not be confused by these 2 subjects. To avoid any possible confusion for now, i will not talk anymore about "MIDI channel mapping".

jjc.

 

 
Message du 02/05/18 21:45
De : "Tom M."
A : "Ceresa Jean-Jacques"
Objet : Re: [fluid-dev] API design: fluid_synth_process()
Post by Ceresa Jean-Jacques
You mean 5.1 (i.e 6 channels)
Sorry i dont' understand this index 5 ?.Does Surround could follow any others know buffers ?
I think each audio channel should receive the full spectrum. fluidsynth shouldnt be in charge of rendering the subwoofer channel. But these are implementation details for surround audio, which is way beyound the scope of this talk. Just wanted to point out the channel layout for fluid_synth_process() if the user wishes to instantiate a surround capable synth.
Post by Ceresa Jean-Jacques
Mapping of dry and effect to output buffers is not easy to solve. I don't think that fluidsynth should impose his strategie.
One might allow buffers to alias. Suppose
"synth.audio-channels" is 1
"synth.effects-channels" is 2
out[0] = out[2] = out[4] = left_buf;
out[1] = out[3] = out[5] = right_buf;
and left_buf and right_buf are zeroed buffers to which fluidsynth would simply add the synthesized audio.
out[0] = left_buf;
out[1] = right_buf;
fluid_synth_process(synth, 64, 0, NULL, 2 out)
Post by Ceresa Jean-Jacques
Note) Also, i think that in the future, the actual internal "MIDI channel to output buffer" hard coded mapping should be replaced by an API.
Shouldnt be the user in charge of properly mapping any multichannel rendered output?
Tom
Ceresa Jean-Jacques
2018-05-03 13:36:34 UTC
Permalink
Hi,

 
Post by Tom M.
One might allow buffers to alias. Suppose
"synth.audio-channels" is 1
"synth.effects-channels" is 2
out[0] = out[2] = out[4] = left_buf;
out[1] = out[3] = out[5] = right_buf;
 

1)Yes, when nout/2 (3) is above rendered synth.audio-channels(1), the idea of expanding the first part of out array(from 0 to 1) into the remainder (from 2 to 5) is fine.

This will allow all load speakers always reproducing something (otherwise (without expansion) load speakers 2 to 5 will be silent)(however to keep equal balancing the user should paid

attention to choose the remainder count of load speaker (4) always multiple of the first part count (2). All this are detail only under the user responsability.)
Post by Tom M.
and left_buf and right_buf are zeroed buffers to which fluidsynth would simply add the synthesized audio.
Yes,

2)Conversely when rendered synth.audio-channels count (i.e 4) is above nout/2 count (i.e 2), the idea to wrap and mix the remainder rendered audio-channels count (from 2  to 3) into the same out buffer that contain the first two audio-channels part count (from 0 to 1) also is fine. This will allow to hear all rendered audio-channels (with no missing MIDI instrument) even when the count of supplied final out buffer (nout/2) is below synth.audio-channels. These idea make fluid_synth_process() tricky and flexible.

 

Also, for the fx rendered channels i was thinking using the same flexibility.

I mean, using unused {nin count, in array} parameter as  {nout_fx count and out_fx array} (as you already proposed) to mix internal rendered fx-channels in out_fx array (the same

way as mixing internal dry audio-channels in out array). This way the user can choose to get the couple {rendered dry audio, rendered fx audio} in a couple of distinct ouput buffers {out, fx_out}.

Note however, if the user choose fx_out to be the same array than out, all rendered audio (dry and fx) will be mixed in a unique array out. Of course the way used by

fluid_synth_process() to do the expansion and wrapping described in (1) and (2) should be clearly documented.

 

details notes:

- nout count (and nout_fx count) supplied by the user should be alway a multiple of 2 (because of rendered stereo audio frame). (This is not relevant when the the user

  intend to produce a surround rendered synth).

- Of course, memory allocation must be avoided in fluid_synth_process().

jjc
Post by Tom M.
Message du 02/05/18 21:45
De : "Tom M."
A : "Ceresa Jean-Jacques"
Objet : Re: [fluid-dev] API design: fluid_synth_process()
Post by Ceresa Jean-Jacques
You mean 5.1 (i.e 6 channels)
Sorry i dont' understand this index 5 ?.Does Surround could follow any others know buffers ?
I think each audio channel should receive the full spectrum. fluidsynth shouldnt be in charge of rendering the subwoofer channel. But these are implementation details for surround audio, which is way beyound the scope of this talk. Just wanted to point out the channel layout for fluid_synth_process() if the user wishes to instantiate a surround capable synth.
Post by Ceresa Jean-Jacques
Mapping of dry and effect to output buffers is not easy to solve. I don't think that fluidsynth should impose his strategie.
One might allow buffers to alias. Suppose
"synth.audio-channels" is 1
"synth.effects-channels" is 2
out[0] = out[2] = out[4] = left_buf;
out[1] = out[3] = out[5] = right_buf;
and left_buf and right_buf are zeroed buffers to which fluidsynth would simply add the synthesized audio.
out[0] = left_buf;
out[1] = right_buf;
fluid_synth_process(synth, 64, 0, NULL, 2 out)
Post by Ceresa Jean-Jacques
Note) Also, i think that in the future, the actual internal "MIDI channel to output buffer" hard coded mapping should be replaced by an API.
Shouldnt be the user in charge of properly mapping any multichannel rendered output?
Tom
Tom M.
2018-05-05 14:30:57 UTC
Permalink
Ok thanks. So we at least have an idea of how to cope with fluid_synth_process(). Implementation might be a little tricky, not sure if if will be ready for the beta. But shouldnt hurt that much, since it wont break existing behaviour either.

Tom
Ceresa Jean-Jacques
2018-05-20 10:16:07 UTC
Permalink
Hello,

 

I have read the proposal (doc API). Thanks for the necessary and useful examples.

I found this understandable but my opinion is probably not objective because this correspond exactly to the previous discussion.

 

Anybody interested by audio rendering to multiples audio outputs should read this. Particularly the possibility of mapping/wrapping and mixing

to the application's output buffers.

jjc

 

 

 
Message du 18/05/18 13:31
De : "Tom M."
Objet : Re: [fluid-dev] API design: fluid_synth_process()
http://www.fluidsynth.org/api-unstable/synth_8h.html#aa8960f664e2b22025a8a72a78a614279
http://www.fluidsynth.org/api-unstable/fluidsynth_process_8c-example.html
Tom
_______________________________________________
fluid-dev mailing list
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Marcus Weseloh
2018-05-18 12:09:23 UTC
Permalink
Hi Tom,

looks very good, thanks!

Initially a was a bit confused with regard to the effects buffers in the
first usage example. I thought that the code would only render the reverb
effects... but then I remembered that the channels wrap around and that
chorus would simply add to the same two channels. So maybe it would be good
to explicitly state this in the example?

Cheers,

Marcus
I have drafted a proposal for a possible implementation of
fluid_synth_process() based on the previous discussion. Anybody interested,
http://www.fluidsynth.org/api-unstable/synth_8h.html#
aa8960f664e2b22025a8a72a78a614279
As it's only a proposal, it hasn't been implemented yet. Please feel free
to give me feedback, whether it's understandable, useful, etc.. There are
http://www.fluidsynth.org/api-unstable/fluidsynth_process_8c-example.html
Tom
_______________________________________________
fluid-dev mailing list
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Loading...