Subject: Re: [linux-audio-dev] PTAF link and comments
From: David Olofson (david@olofson.net)
Date: Tue Feb 04 2003 - 03:11:22 EET
On Monday 03 February 2003 23.52, Tim Hockin wrote:
> From: David Olofson <david@olofson.net>
>
> > * Three states; created, initialized and activated.
> > This may be useful if plugins have to be instantiated
> > for hosts to get info from them. Why not just provide
> > metadata through the factory API?
>
> What happens during 'created', and what transitions to initialized?
Seems like you can start talking to the created plugin, basically.
That is, whan you get is something barely alive, that can only handle
some basic communication.
I guess the idea is that the host can give the plugin some info before
asking the plugin for the "meta-data". There is a comment abot
keeping the creation and destruction operations low cost, whenever
possible, since hosts may have to scan loads of plugins at startup...
Compared to XAP, it looks like PTAF has a plugin state where we have
the "plugin factury/meta-data level", sort of; ie the next two states
corresponds to the two states we've proposed for XAP.
> > * Bypass mode seems to be a good idea for stereo->surround.
>
> if your plugin just does some sort of spatialization (mono to
> stereo, mono to 5.1, stereo to quadra, whatever), then what does
> bypass mean?
The simplest and/or most "straight" mapping you can figure out. Say,
map 2 ch stereo to front-left and front-right or best match in
whatever surround format you're outputting.
> I lik ethe idea of having a standardized bypass
> mechanism, but I want to know what it means in that case?
Well, you can't know *exactly* what will happen, as there might not be
an obvious solution...
> If your
> next plugin in the chain expects 6 channels, you can't easily
> remove a 2->6 channel converter from the net. So does bypass mean
> nothing for that plugin?
On the contrary, this is exactly when you need the bypass feature,
since the host can't just rip the plugin out temporarily.
> If there is some notion (example: stereo
> can be front-left and front-right in a 5.1, with the rest silent)
> of how to translate, then it is OK, but can that be assumed?
Well, yeah. It's not a strict definition, but rather "do something
that sounds as if the plugin is doing nothing."
> > * Assuming that audio and sequencer stuff is in different
> > threads, and even have plugins deal with the sync sounds
> > like a *very* bad idea to me.
>
> ick
>
> > * I think the VST style dispatcher idea is silly. A table
> > of function pointers, with a bunch of reserved NULLs
> > would be simpler, faster and just as extensible for all
> > practical matters.
>
> There is a certain cleanliness to not having to check for NULL
> methods in the host, but I think that it does not warrant an extra
> level of indirection. structs can grow cleanly. Stick with that,
> I say.
Yeah, that should work. Just make sure structs *can* grow in the first
place, and then it's just a matter of checking versions and pointers.
(Which can all be done by the host SDK for lazy/effective host
writers, of course.)
> > * UTF-8, rather than ASCII or UNICODE.
>
> I don't know enough about localization - how does this affect
> standard use of printf?
>
> char *s = plugin->name;
> printf("NAME: %s\n", s);
It doesn't, basically, as what you have physically is just normal 8
bit strings with some extra control codes in them. It matters when
something actually tries to render the strings, though, and that's
the point. UNICODE without using 16 bit "char", basically.
> > * Hosts assume all plugins to be in-place broken. Why?
> > * No mix output mode; only replace. More overhead...
>
> Can of worms - more below
>
> > * Buffers 16 byte aligned. Sufficient?
>
> what is the point aligned at sizeof(double) should be sufficient.
> If we want to impose more restrictions, they should be well
> justified.
Yes. (See below.)
> Or they should be left to the host.
Yes. In PTAF, it seems like the point is just that plugins can assume
this alignment, and hosts are supposed to guarantee it.
> But I don't know
> enough platforms to know if this matters to some. Anyone have
> better feedback?
The newer SIMD extensions work with 128 bit words AFAIK, and 128 bit
memory busses are getting more and more common... That's where the 16
bytes are coming from, I think.
> > * Audio quality control. (Nice scalability feature.)
>
> elucidate? I haven't read the spec yet...
Uhm... Right! :-)
It's the same kind of scalability feature I have in Audiality (not
that *that* would clear anything up...), where it among other things
allows you to use various types of interpolation for the voice
resampling. You can't do adaptive cubic/linear-oversampled on an old
Pentium, but you can just switch to linear or even nearest to save
cycles.
PTAF uses float, subdivided in four ranges; [0,.25], [.25,.5],
[.5,.75] and [.75,1], where the lower two ranges are for real time
use, and the higher two are for off-line use.
> > * Plugin input->output latency.
> > * Host process return->audible output latency.
>
> in my notes for XAP already
>
> > * Tail size. (Can be unknown!)
>
> in my notes, in a different way, but this may just be simpler
Yeah... It's guessing and various hacks anyway for many plugins, so
you can't require much, and I don't think there's much point in going
below block granularity. It gets hairy, and I think few hosts and
plugins will make full use of it even when it's theoretically
possible. Just too much work, and it actually *costs* cycles when you
need them most: when all plugins are running full throttle.
> > * Process mode: Mixed/RT/Off-Line.
>
> do we really want anything like this? I have a 1-10 'quality'
> level.
This is different from quality. Consider a disk sampler with a butler
thread. That just won't run off-line, because the audio part isn't
driven by a stable time base, and it won't do the right thing (ie
*block*) when the audio part runs too fast. Similar problems with
animated GUIs and the like.
If plugins just get a hint about what's going on, they can deal with
this quite easily, or just avoid doing silly things.
> > * Plugins send events specifically to the host...
>
> We already have something like this - if the host is going to draw
> UIs, it needs to snoop event traffic.
Yes, but that's not explicit sending to the host, as in using a
special API for only that, as they do in VST, and apparently in PTAF.
Plugins just have control ins and outs; that's all. Plugins can't
even tell whether these are connected directly to other plugins in
the same thread, or to other threads or processes through host
managed gateways. I don't see a need for duplicating major parts of
the API, and introducing tonnes of special cases in plugins, just for
this kind of stuff.
> > * Parameter sets for note default params? Performance
> > hack - is it really worth it?
>
> explain more?
Well, it's rather similar to our voice control system. Voices/notes
have a bunch of controls (generally continous, just like ours), which
have default values that are loaded when you create a voice context.
In PTAF, that's done with the same event that starts the note... They
have two note-off's though; one for the actual note-off, and one to
release the voice context, when there are no more control changes to
send.
Anyway, the defaults would normally be hardcoded into the synth or
something - but as I understand it, PTAF has some kind of "parameter
sets" or something that you can throw at voices instead of flooding
them with the same control data for every note.
That is, if the sender is going to play lots of notes where most voice
controls are identical, but not the voice defaults, the sender can
build a parameter set and refer to that instead, or something like
that...
As to XAP, I would prefer a light weight hint based solution for this.
Allow plugins to specify a range of Channel controls that map 1:1 to
the Voice controls on that Channel. Whenever the synth creates a
voice context, it just grabs the defaults from those Channel
controls, instead of using some internal constants.
You get two things:
1) Senders/hosts can change all the default voice
controls as desired, using normal Channel control
events.
2) Bonus: The Default Voice Controls automatically get
into presets, since they're just normal controls.
> > * Why have normalized parameter values at all?
> > (Actual parameter values are [0, 1], like VST, but
> > then there are calls to convert back and forth.)
>
> makes connecting any arbitrary output to any input easy.
Yeah, but is it really that useful? Ok, VST seems to survive with the
same restriction, but I'm not sure if it actually solves more
problems than it creates.
Anyway, *if* one is using [0, 1], the conversion calls that ptaf have
(but VST lack, AFAIK) are essential, of course.
> > * The "save state chunk" call seems cool, but what's
> > the point, really?
>
> Well, it is nice to be able to have a plugin store some random gunk
> for a preset. This could just be a raw data block, except that we
> do not have readable controls - we've been expecting that the host
> just remembers the value of controls.
Yeah, but that's a different thing. This is explicitly for stuff that
isn't accessible through any other interfaces.
For storing presets, I don't see a reason to have anything like full
plugin side preset load/save. I moves quite a lot of non-portable
code into plugins for no real reason.
Just have the host store the control values, and use string or raw
data block controls for anything that simply won't fit in standard
controls. The host collects the data through the standard control
API, does the file I/O and all that.
> From: Steve Harris <S.W.Harris@ecs.soton.ac.uk>
>
> > > * Hosts assume all plugins to be in-place broken. Why?
> >
> > It doesn't really matter what the default is as long as you can
> > override it. That way is probably safer.
>
> Do we want to do this for XAP? I'd kind of hoped that XAP would
> dictate that all plugins must be in-place safe.
Yeah, that would be reasonable, I think. In the few cases where
plugins can't be in-place safe (or the author is too lazy), we even
have a nice, cache friendly solution: The audio buffer pool. (A LIFO
stack of preallocated buffers. Optimized hosts will probably use one
anyway, instead of fixed buffers all over the net.)
> From: David Olofson <david@olofson.net>
>
> > You don't necessarily *have* to implement both. Even the
> > primitive FX plugin API of Audiality have these variants:
> >
> > void (*process)(struct ADY_plugin *p,
> > int *buf, unsigned frames);
> > void (*process_r)(struct ADY_plugin *p,
> > int *in, int *out, unsigned frames);
> > void (*process_m)(struct ADY_plugin *p,
> > int *in, int *out, unsigned frames);
> >
> > ...and you only *have* to provide *one* - any variant will do. If
> > you don't provide all of them, the host "SDK" will emulate the
> > others using the ones that are provided.
>
> This seems like a lot. Where is the performance really going to
> go?
Yeah. Only the last two are relevant, really, and maybe even two is
more than we need...
> Unlike the above API, there isn't necessarily a direct input-output
> mapping.
That's a good point. That's why the inplace-with-single-buffer-pointer
version is irrelevant to "real" plugin APIs.
> What types of plugins use which modes?
>
> Instruments: have no inputs (in general) and overwrite output
> Effects with #ins = #outs: overwrite their output with:
> (input * dry) + (fx * wet).
They don't really have to scale the input... If the idea is to
optimize the inplace case in a configuration where you're doing "in
bus" wet/dry mixing, you can just scale the buffer before calling the
plugin. No extra buffers needed, and that's the big win, I think.
> We can standardize a wet/dry gain control pair. But this becomes
> something every plugin needs to provide. Uggh.
It *could* be optional... The host SDK would deal with plugins that
don't bother to implement those controls.
Anyway, the problem with this is that there doesn't have to be any
obvious or known relation between inputs and outputs. You can still
do inplace processing on some input/output pairs, but then, what are
you actually doing?
In-place replace is obvious (reuse the buffers becaues we don't need
the audio after the plugin has read it), but in-place add or mix...?
> Better if they are just plugin specific. If a Plugin doesn't
> provide wet/dry control, a simple send plugin can be inserted,
> right?
Yeah...
> Which leaves us with LADSPAs run and run_adding. Which I can only
> see useful for the return of send effects, and that can be handled
> with a simple mixer plugin, but maybe shouldn't.
Well, the problem with that case is that you need an extra buffer to
do that mixing. If the plugin has an adding mode with a standardized
gain control, you can skip both the intermediate buffer and the extra
mixing stage, and then it starts to matter.
> From: Sami P Perttu <perttu@cc.helsinki.fi>
>
> > > ...and you only *have* to provide *one* - any variant will do.
> > > If you don't provide all of them, the host "SDK" will emulate
> > > the others using the ones that are provided.
> >
> > I think this is bad. There should be just one process() function,
> > which could be given two gain values, one for previous output and
> > another for the plugin's own output. Plugins would do
> >
> > out[i] = previous_gain * out[i] + gain * myoutput;
>
> Then the gain (wet/dry gain is what you're saying) is not
> automatable. What's the point?
Why not? They could just be normal ramped controls.
out[i] = previous_gain * out[i] + gain * myoutput;
previous_gain += dprevious_gain;
gain += dgain;
That's all there is to it, apart from moving those variables into the
array of controls, so they get hooked up with the RAMP events.
> > PS Your point about static metadata is well made, but some items
> > that could be considered as metadata (eg. port ranges) may depend
> > on system varaibles (sample rate etc.), though that should
> > probably be discouraged.
>
> Well, there also needs to be a way for a plugin to wrap other
> formats, and change all it's metadata. I imagined that a plugin
> couls tell the host that it needs to be re-examined -
> XAP_EV_GESTALT - or something.
Yeah.
> > Yeah, that's what I thought at first. However, if you track the
> > previous_gain and gain controls, you can select between a number
> > of optimized inner loops internally. What you get is basically
> > the same thing as a bunch of different callbacks, except that
> > it's not part of the API, and plugins can handle it any way they
> > like.
> >
> > Oh, you *do* have to check the control events for those two
> > controls, of course.
>
> do we really want to force this into all plugins? I'd rather see a
> send effect that handles this. Not that run_adding isn't needed -
> it makes the eventual ending just a bit simpler, but I am dubious
> of it's real value.
>
> Simpler is better.
Yeah. run_adding() is indeed nothing but a performance hack. Question
is if we really need it these days. I'm not saying that we'll ever
have enough cycles to just waste them, but focus is shifting
continously. Things are getting higher level.
VST is a really rather old API, and that memory bandwidth in new PCs
have increased quite a bit even after LADSPA was finalized... Not
saying that run_adding() makes no difference to anyone *now*, but in
a few years, it'll probably just be a source of annoyance.
> > That said, I still think it seems easier to just provide a few
> > different callbacks of which plugin authors can pick one or more.
> > A
>
> Can you elucidate on what different ones are needed, bearing in
> mind that buffers are not passed to the run method, but rather
> connected to ports beforehand?
It doesn't really matter when you pass the buffer pointers. Even if
you do it at connection time, you can pass the same buffer to a lot
of plugins in a chain, since that's pretty much what a per-plugin
dynamic buffer assignment system with a LIFO stack would do anyway.
Either way, I can't really think of more than two variants; process()
and process_mix(), where the latter would have a standardized gain
control, to avoid using both an extra buffer and a mixing loop where
chains end and the like. I think that's about it...
> > However, your question gives me an idea: Transform the
> > dispatchers into functions that just return direct function
> > pointers by index. Dead simple, and you can deal with unknown and
> > unimplemented functions in any way you like. You have to ask for
> > all the call you'll need at some point (say, right before
> > activating the plugin, so it has a chance to select specialized
> > versions, if any), but it's cleaner and simpler than dynamic size
> > arrays on both sides, I think.
>
> I don't mind this, but I don't know if I see the reason it is
> NEEDed.
Right. I don't think it is, really. Just don't fix the size of structs
in the API, and you have unlimited expansion possibilites *and* a
clean API.
> > > This seems like it would subsume plugin based settings saving,
> >
> > Actually, I don't think it's related at all. Plugin state is the
> > state of voices, contents of delay buffers and other internal
> > stuff; ie everything that *isn't* available as parameters or
> > controls.
>
> It COULD. Rather than the host saving formatted data for controls,
> we could certainly just have the plugin do it, and save per-plugin
> opaque blocks. No reason to do that if we can accurately track
> control values.
Right. And it seems that doing this in plugins bring nothing but
trouble and extra work for plugin authors in the APIs that do it that
way...
> However, I really do see a potential need for the
> plugin to store internally generated data in an opaque manner. A
> Raw data block control works great for this
Yes... We have strings and raw data block, it seems.
>- except we don't have
> any way to READ a control. Just write.
Yeah, that's an issue. You could use outputs and "request" events, but
then these request events make it all non-standard anyway.
OTOH, plugins have to assemble this data somehow, as it's probably
rarely raw dumps of internal structs. (That tends to be very flaky
and non-portable.) Thus, there's no data you can just read; you have
to get some code in the plugin put it together for you.
Or do you?
Looking at normal controls, you never have to read the inputs, since
you can track whan is sent to them when you need to. Why wouldn't
that apply to these raw data blocks as well? Remember that the DSP
plugin (that receives and uses the data) and the GUI (which generates
the data) are separate plugins, using basically the same API. The
host would just intercept and store the raw data blocks like any
(other) control values, when they come from the GUI. Or, when you
load a preset, it would send the blocks to the inputs of both the DSP
plugin and the GUI.
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
--- http://olofson.net --- http://www.reologica.se ---
This archive was generated by hypermail 2b28 : Tue Feb 04 2003 - 03:15:37 EET