Subject: Re: [linux-audio-dev] XAP spec - early scribbles
From: David Olofson (david@olofson.net)
Date: Wed Feb 05 2003 - 13:07:13 EET
On Wednesday 05 February 2003 04.58, Tim Hockin wrote:
[...]
> 0.3 Terminology
[...]
> * Host
> The program responsible for loading and controlling Plugins.
Well, you know my opinion about what controls plugins.... ;-)
For those who missed it: My POV is basically that the "sequencer" is
not necessarily an integral part of the host. It might as well be a
plugin. My motivation is that assuming this makes the control API a
lot cleaner and generic WRT event processing and the like. The idea
is that you shouldn't need special API extensions to do "MIDI plugin"
style event processing, and more importantly, that the API should not
force hosts or other entities to touch events when one plugin is
supposed to control another. Connections are by all means direct,
except when the host explicitly wants to eavesdrop.
[...]
> * Control:
> A knob, button, slider, or virtual thing that modifies behavior of
> the Plugin. Controls can be master (e.g. master volume),
> per-Channel (e.g. channel pressure) or per-Voice (e.g.
> aftertouch).
I think it might be a bit confusing to drag GUI widgets and stuff in
here... We don't deal with that at all on this level. (The GUI
plugins more like an application of the API - which I can't wait to
start messing with, as you might have guessed! ;-)
Anyway, a Control is basically an abstract object that has a state
containing a value, and (optionally) a slope that controls ramping of
the value. Events can be sent to the Control to set it's value, or to
set up linear ramping of it.
Too tired to write something comprehensible now, but you get the
idea... :-)
[...]
> * EventQueue
> A control input or output.
Confusing. Sounds too much as if an EventQueue would be more or less
synonymous to a Control - which is generally very far from the truth.
A single EventQueue can drive an entire polyphonic synth, if the
author doesn't see any advantage in having events dispatched directly
to internal units as they are sent.
> Plugins may internally have as many
> EventQueues as they deem necessary. The Host will ask the
> Plugin for the EventQueue for each Control.
Yep. The effect of this might be cool enough to deserve a comment;
"The point with this design is that it allows Plugins to have input
Events delivered directly to the EventQueues of it's internal units,
which means that there is rarely a need for actively sorting or
dispatching Events. Each unit will get only the Events meant for it,
and Events will be in timestamp order, as always."
> FIXME: what is the full list of things that have a queue?
> Controls, Plugin(master), each Channel?
Why make a list? This is essentially left to plugin authors to decide.
All we can say is that a plugin instance will need at least one
EventQueue for input, but that's about it.
There could be private EventQueues internally, invisible to the world
outside. (Better idea than rolling your own, provided they do what
you need.)
There will never be "output EventQueues", since you're always sending
directly to the connected inputs, as far as plugins are concerned.
Whether you're actually sending directly to another plugin, or to a
host managed sort/merge proxy queue is pretty irrelevant to the API
and plugins, since it's a host implementation thing.
[...]
> All XAP audio data is processed in 32-bit floating point form.
> Values are normalized between -1.0 and 1.0, with 0.0 being silence.
I think "normalized" is the wrong term here, since it can't be more
than a 0 dB reference. There's no useful way to actually *normalize*
audio data.
[...]
> 1.2 Events
>
> Almost everything that happens during the ACTIVE state is
> communicated via Events. The Host can send Events to Plugins,
> Plugins can send Events to the Host, and Plugins can send Events to
> other Plugins (if they are so connected).
Though it's still not clean what events would be sent explicitly from
the host to a plugin and vice versa...
All I can think of right now is that worker callback API I proposed
for file I/O, heavy waveform rendering, dynamic memory allocations
and other inherently non RT safe stuff. You'd make a function call to
the host telling it to call a specific function from some suitable
thread (which could mean it's just called directly), and then you get
an event from the host when the function returns.
[...]
> 1.3 Channels
>
> //FIXME:
Rather similar to MIDI channels, but that's about as much as we seem
to agree on so far. This is closely related to I/O layouts and the
metadata to describe those, and we have some work left to do in that
area.
> 1.4 Ports
And connecting an audio port means handing it a buffer pointer, among
other things? That is, buffer pointers aren't changed frequently, and
never silently.
I *think* this should do, since it does not conflict with reuse of
buffers. Hosts can just figure out optimal buffer usage patters when
building nets. Right?
[...]
> 2.0 The Plugin Descriptor
[...]
> (*) The descriptor for wrapper Plugins may change upon loading of
> a wrapped Plugin. This is a special case, and the Host must be
> aware of it. //FIXME: how?
That's messy stuff... It means that descriptors become somewhat
dynamic data belonging to an *instance*, rather than static data
related to a class. This kind of suggests that that third state of
PTAF is a pretty good idea, since it automatically allows each plugin
instance to mutate and become an instance of another class - which is
exactly what we need here.
[...]
> 2.2 State Changes
>
> As mentioned above, Plugins exist in one of two states - ACTIVE
> and IDLE (well, three if you count non-existance as a state).
4, if you count "CLASS" as a state - or whatever we should do to
handle this "mutating plugins" stuff...
[...]
> 2.2.1 Create
>
> A Plugin is instantiated via it's descriptor's create() method.
> This method receives two key pieces of information, which the
> Plugin will use throughout it's lifetime. The first is a pointer
> to the Host structure (see below), and the second is the Host
> sample rate. If the Host wants to change either of these, all
> Plugins must be re-created.
I'm afraid this won't work for all hosts. There are situations where
you might want to run a plugin under a fake copy of the host, in
another thread, to perform non RRT safe operations without disrupting
the rest of the network.
This is hairy stuff, though, and I'm thinking that it would be nicer
if all plugins were *required* to be RT safe - even if they do it by
sending unsafe operations off as worker callbacks. That won't
guarantee real time response to the offending events, but it'll keep
plugins from stalling the audio thread.
Of course, asynchronous programming isn't exactly trivial either, so
this doesn't exactly eliminate the problem...
[...]
> 2.2.3 Activate
>
> From the IDLE state, a Plugin can be changed to the ACTIVE state
> via the activate() method. Passed to this method are two arguments
> which are valid for the duration of the ACTIVE state - quality
> level and realtime state. The quality level is an integer between
> 1 and 10, with 1 being lowest quality (fastest) and 10 being
> highest quality (slowest). Plugins may ignore this value, or may
> provide less than 10 discrete quality levels. The realtime state
> is a boolean value which is boolean TRUE if the Plugin is in a
> realtime processing net or boolean FALSE if it is not realtime
> (offline).
Sounds good.
[...]
> 2.4 Port Setup
[...disable etc...]
Nice. Can be used for "optimized silence", of course - which leads to
a question: How about *outputs*? Plugins that keep track of whether
or not they're actually generating audio would need a way to mark
each output as "audible" or "silent", before returning from each
process/run() call.
> 2.5 Controls and EventQueues
>
> //FIXME: kinda needs Channels
> In order to be of any use, a Plugin must provide Controls.
> Control changes are delivered via Events. Events are passed on
> EventQueues. Each Control has an associated EventQueue, on which
> Events for that Control are delivered.
Yes.
> In addition, there is an
> EventQueue for each Channel and a master Queue for the Plugin.
Why make this explicit? Is there *any* case where you would send
events to a specific Queue, rather than some target you're connected
to?
Either way, I think it's a bad idea to enforce that these per-channel
and master Queues should actually be physical Queues. The problem
with it is that it forces plugins to sort/merge and/or dispatch
events internally, unless their internal structure happens to match
the master + channels layout.
Provided that cookie addressing is used for *all* events (except
possibly master events, although making them special seems evil to
me), there's never a need to use Queues as part of the addressing.
> The Host queries the Plugin for EventQueues via the
> get_input_queue() method. In order to allow sharing of an
> EventQueue, the get_input_queue() method also returns a cookie,
> which is stored in each Event as it is delivered. This allows the
> plugin to use a simpler EventQueue scheme internally, while still
> being able to sort incoming Events.
I would prefer calling it get_control_target() or something, given
that what you actually get is not a Queue, but rather an "atom"
containing a Queue pointer and a cookie. Each of the two fields of
this "atom" (XAP_target) is completely useless on it's own, but
together, they form the complete address of an input Control.
> Controls may also output Events. The Host will set up output
> Controls with the set_output_queue() method.
How about connect_control_output() or similar? Again because queues
have very little relevance on this level, and because "connect" is
more accurate than "set", IMHO. (Which is not the case with
get_<whatever>(), since that doesn't affect anything; it just asks
for some info.)
Another thought: Would plugins want to know when their inputs get
connected and disconnected?
> 2.6 Processing Audio
>
> A Plugin which has been activated and properly set up, may be
> called upon to process audio. This is done through the
> descriptor's run() method. This method gets the current
> sample-frame timestamp as an argument. In this method the Plugin
> is expected to examine and handle all new Events, and to read from
> or write to it's Ports.
>
> This method may only be called from the ACTIVE state.
It must be explicit that you're supposed to process events for an
audio frame *before* processing the audio data for that frame. If it
isn't, plugins will behave differently in ways that might break some
hosts.
More importantly, when do you stop processing events? There are two
alternatives:
1) Process only events that affect audio frames that
you're supposed to process during this block.
2) Process as many events you can, without skipping
into the next block.
The difference is very important, and it becomes rather obvious when
you consider what will happen if you ask run() to process 0 frames.
In case 1), this will do exactly nothing, while in case 2), in will
cause any events for the next audio frame to be processed.
In that light, 1) seems essentially useless, while 2) might be really
rather handy. Either way, 2) doesn't break anything, so we might as
well define that as the correct behavior.
If this seems incorrect, consider these questions:
* When are events actually handled; *between* audio frames,
or *on* audio frames?
* Do the two "stop rules" differ in the way timestamped
events affect generated audio?
[...]
> Plugins can assume that VVIDs are at least global per-Plugin.
> The host will not activate the same VVID in different Channels at
> the same time.
Well, as usual, I'd just like to whine some about it being assumed
that the sequencer is an integral part of the host. ;-)
It could be an event processor or a phrase sequencer you're listening
to. Not that you'd care when you're a plugin in the receiving end,
but things get weird if you're about to implement a plugin that
outputs synth control events, while in the API docs, it appears that
only hosts can do such things...
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
--- http://olofson.net --- http://www.reologica.se ---
This archive was generated by hypermail 2b28 : Wed Feb 05 2003 - 13:07:16 EET