Subject: Re: [linux-audio-dev] big picture, for a moment
From: David Olofson (audiality_AT_swipnet.se)
Date: ma loka 11 1999 - 22:13:24 EDT
On Mon, 11 Oct 1999, Paul Barton-Davis wrote:
> I think we need to step up a level or two, and try to clarify the big
> picture again.
Yes, that's a good idea, I think... :-)
I also think that your description indicates that we're close to
getting a very nice design together. I don't agree fully with all
parts, but it seems rather close.
> >From my perspective, I am now agreed that we should use an event
> system similar to the one David has proposed. I agree that we should
> use a constant frame/sample count argument to plugins, and let them
> deal with events in any way they choose. I agree that we need 64bit
> timestamps. External event sources (MIDI, audio cards, mice,
> keyboards, X servers, etc.) will be handled by the semantic equivalent
> of "sensor threads" that are partly autonomous, and partly dependent
> on the engine thread.
I would like to add that it's also possible to make device drivers in
the form of plug-ins, which makes a lot of sense in the kernel. That
is, unless Jaroslav dislikes our API too much, the ALSA drivers could
gradually turn into plug-ins, running in a basic kernel implementation
of an MCS plug-in host. That would make it pretty easy for him to
move code around in the driver architecture, and even move parts to
user space if desired. Would that be a nice thing for the 3D sound
rendering and similar stuff, perhaps?
> I think that David agrees that we will not use
> shared memory directly to communicate with "clients" (other
> processes), but instead will use "sensor threads" that will make
> shared memory function like any other event source. Note: an
> implementation may or may not use actual threads to implement this,
> but the API will be defined so that threads *can* be used.
Yes, the power of a carefully designed API; inherent abstraction on
the binary level, so that the API implementation can do pretty much
what it wants without uggly workarounds.
> Where David and I currently disagree is the relationship between
> plugins and event ports. I think of a plugin that is introduced to the
> system coming into a world of many possible different event ports, and
> having a clear idea of the kinds of things it wants to pay attention
> to. Those may change as a result of user interaction, but I see the
> plugin as "subscribing" to various event ports, and unsubscribing at
> other times. Plugins may also be (un)subscribed to ports as a result
> of user interaction and/or automation playback. The engine provides a
> way to locate a port given its address, so that plugins can subscribe
> to its ports of interest. Objects that generate events have no idea
> who is subscribed to their ports. All events are timestamped using a
> time base maintained by the engine, but usable without having to
> preempt the engine thread (if it is indeed a distinct thread).
...while I view a plug-in as a black box with a number of inputs and
outputs, in the form of audio ports and event ports. The average
plug-in would have >=0 audio inputs, >=0 audio outputs, one event
input, and possibly one event output. During initialization, the
plug-in will tell the host what events it understands (little more
than info for the UI), and if it has an output port; what events it
might send through it. The plug-in doesn't care (or know) who listens
to it's output, nor does it know where the events it recieves come
from. _It processes events in the same manner as it processes audio;
as an input -> output black box._
In order to connect event ports in a useful way, the engine may
remap events (of compatible types), so that for example, a plug-in
sending "float #1" can be set up to control another plug-in that
listens to (among other things) "float #5". Of course, if "float #2"
is to be connected to "float #2", no remapping is needed! OTOH, if
"float #2" is to be sent to a third plug-in, the engine will start to
_reroute_ those events to the event port of that plug-in. Neither of
the plug-ins need to know anything about this; they just do their job.
The remapping can be carefully optimized in the inner workings of the
engine implementation for maximum performance, or handled by a basic
event router in the MCI "Engine Toolkit" library. As this is kept
entirely separate from the plug-in side of the API, it's even
possible to build special event filters and operators right into the
engine, so that users that want to handle ektreme event traffic can
do so with minimum overhead. (However, the normal and *flexible* way
of adding detailed event processing would be using event-only
plug-ins, much like some systems use special MIDI plug-ins.)
Finally: What about event routing overhead?
Well, compare that to the overhead involved when handling the number
of event ports needed to get the same flexibility.
Input from multiple senders:
* The engine has to merge the events into one list,
and they must be sorted according to the timestamps.
* _ALL_ events that are output from every subscribed
source must be merged in, as the engine doesn't
know which ones the listening plug-in cares about.
* Event remapping *still* has to be done, as events
from multiple sources *may* collide otherwise.
* The plug-in not only has to subscribe to ports to
get events, it also has to keep track of how the
"anti-collision" remapping affects the events it
wants.
* The alternative is to check multiple input ports,
which would pretty much kill the event system's
efficiency, and make the code quite complex.
Output to multiple destinations:
* _ALL_ events will have to be copied to _ALL_
destinaiton ports by the engine, as the sending
plug-in has no idea who's listening to what. And
neither does the engine...
* The engine would have to remap all events, unless
every destination has a dedicated port for the
sending plug-in.
Well, my design has a few problems as well, of course... For example,
event remapping. But by structuring the event codes (32 bits in my
examples so far) in a sensible way should reduce that to one or two
levels of lookup tables in the optimized implementation. Perhaps
there are even better ways? I think we might find interesting
solutions in IP routers and similar systems. I bet there are
interesting parts of The Kernel to study!
Now, lets find the flaws in my version! :-)
> Next are some ideas that we have not talked about.
> -------------------------------------------------
>
> I propose removing the engine from any involvement in event
> distribution. We can do this because event distribution does not
> involve data copying.
This is true for point-to-point connections, while subscribing to
multiple event sources put you in a network...
> The engine is responsible for setting up
> connections so that it can adequately grok the nets that the
> interconnections create.
Yes.
> But event sources simply post events to their
> ports, and the port subscribers process the events. The engine is
> responsible for marking the last event on any given port that should
> be considered during a plugin's process() call.
Very handy... However, I wonder if it's really a win, performance
wise. How much does it cost the engine to put that end-of-cycle
marker in, compared to the plug-in checking the timestamps?
What I'm thinking about is that it's not necessarily true that there
are no events after the last one to be processed during the current
cycle. There may be events to be processed later on... This happens
when interfacing between sub nets with different buffer sizes, and
when communicating accross threads.
> It is also responsible
> for removing all events up to and including that last event at the
> appropriate time, and doing appropriate games in memory with them.
Yep. (Better make sure to grab as much info as possible whenever the
engine's close to the events, or there may be a lot of traversing
overhead.)
> Plugins and clients are different.
Yes, but slightly less different, IMO.
> A plugin in a precompiled object file which can be dynamically linked
> into any program that supports the engine-side of the API. A plugin
> can subscribe to any event port known to the engine, register new
> event ports, manage its own GUI (which will run in its own thread) and
> consider itself closely coupled to the engine.
As long as the GUI is also in its own *file*, OK. I can load ELF into
kernel space and similar weird places, but I don't feel like cleaning
out "irrelevant" code on the fly...
(See above re the subscription.)
> A client is a standalone program that uses the engine to route data,
> or routes data for the engine. Clients do not have access to ordinary
> event ports,
Why? That's *exactly* where the intermediate FIFO hidden behind the
shadow ports come in handy. If sensor threads can do it, so can
clients, *and*, it makes them a lot more similar to plug-ins. A
client is to MCS what a plug-in is to an engine... To some extent at
least; clients can do lots of cool stuff that plug-ins cannot.
> but instead receive data streams and/or requests for data
> from the engine.
How does it recieve requests, if not through an event port? Using the
client/server model proposed by Benno, a client behaves pretty much
like a plug-in - it just sleeps instead of returning to the host.
However, it might do other tricks as well, and it can even
completely drop out of the communication for a while - and then jump
back in when it has something to say. The cycle is flexible and
optional, as opposed to the plug-in case, but the concept is still
there. Why break it?
> To make this point clearer, if you write an audio
> sequencer as a *client* (i.e. it sends its audio data to an engine,
> not directly to an audio device) it cannot receive events (say, GUI
> changes or MIDI stuff) from the engine. If you write it as a *plugin*,
> then you can.
The engine *is* a client to the *MCS*, as I see it. Or, call it a
"local domain"; a performance optimized model of the MCS. Hang a
gateway onto the client interface, and your local processing is all
of a sudden *publicly* available to any MCS client with permissions!
Plug-ins in your engine can be connected to things in another engine
inside another application. THAT's my idea of integration.
> However, even if the client case, you can still use the MCS library to
> manage events within your own process space, and you can also host
> plugins that use the API. This is the "anyone can write an audio
> server" approach :) More significantly, by doing this, you can almost
> certainly conveniently package the functionality of your application
> in a format that can be a plugin.
Not using event ports as a natural part of the client API makes that
a bit harder, I think... But other that that; yes, this is the idea
with the uniform API. However, I want to be able to get that
*functionality* by hooking things up at run time as well. "Plug" an
application into another, without requiring an extra API layer. A
client adaptor plug-in could be used, in case that simplifies things.
BTW, hint: sensor threads...
> Likewise, you might write an application that routes audio over a
> network using some sensible protocol.
Yes, but why not passing along events as well? IMO, they're not very
different from audio data - or video data for that matter - on this
level.
> Individual users of the MCS system would need to decide if they
> preferred plugins or clients for certain purposes.
Yes, basically;
speed & latency ==> plug-in
incredible flexibility ==> client
> We have not talked about how to address ports. Strings are easy, but
> speed daemons will prefer integer values formed via some scheme that
> encodes the port's "name".
With my model, it would probably make sense with integer port
handles, and strings to initially look up ports. OTOH, reconfiguring
the event routing in a heavily optimized engine might be relativily
expensive... (IMO, changing the processing net - the event routing
included - in real time with guaranteed latency isn't such a high
priority, as it can't be done in the middle of a cycle anyway. But
why waste CPU time if it could be avoided?)
> However, the description I have given above, I think that this is an
> API that I can and would use for all my audio/MIDI programs.
It's a nice design, but I accept no limitations! >;-D
//David
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:27:13 EST