Subject: [linux-audio-dev] Plug-in API design decisions; please consider
From: David Olofson (audiality_AT_swipnet.se)
Date: su elo 29 1999 - 08:26:20 EDT
Hi!
Had a few thoughts, but I don't have much time today, as I *really* need to get
this darn song together. (Sometimes, I just don't love music all that much...)
.-------
| Are we concentrating fully on audio-only, or
| should we take other forms of data into account?
`-------
I think this is very important to think about. There's more places than auido
processing where you could use a standard streaming plug-in API. The most
obvious one is probably video, and IMO, it seems pretty similar to audio
processing if you define one frame as a video frame, instead of an audio sample.
What about defining data format descriptors as a small structure or bit
pattern, rather than a single dimension list of constants? This would separate
the low level details from the physical format that transport layers need to
know about, and means that an engine could stream data that it doesn't
understand, as long as plug-ins to translate are available. "Stream video along
with your audio data in any audio sequencer - no video support needed. Use the
system's multimedia viewer plug-in." Cool or pointless?
What I'd want in a format code is
Low level:
Format ID
Version/details/flags...?
High level:
Channels (per frame)
Frame size (in bytes)
Further, a way of implementing variable buffer sizes would be very useful in
some cases. (Compressed data...) Setting frame size to FRAMESIZE_VARIABLE and
starting each frame with an unsigned long specifying the size would be a simple
interface for engines to handle. The engine doesn't have to know what's being
streamed - it just feeds the data between plug-ins. The engine's interface to
the outside world is the only place where data formats really matter.
Sidenote: Why do I like this?
Well, not only does it mean that host application developers have less stuff to
support - they could actually support *everything* by just implementing a basic
generic data streaming engine and using plug-ins for the actual work; it also
means that the client/server architecture I'll use for Audiality will become a
lot more usable. The engine doesn't need to understand the data all plug-ins
are using. It could handle (split/merge/convert...) the most frequenctly used
formats internally for performance reasons, but doesn't need to, as plug-ins
can take care of it, as long as the data can be streamed correctly.
I never liked the host application approach. Engine plug-ins should be a
shared system resource, and there are many reasons for that.
* It makes it a lot easier for applications to share hardware
resources in an efficient way. Important if multiple
applications want to stream from/to disk at once.
* Synchronization is done through the engine, more or less
for free.
* The engine takes care of synchronizing multiple cards and
other tricky stuff. With sub-sample precision if you have
an RTLinux engine in the system.
* Engine optimizations and bugfixes benefit to all client
applications.
* Any application will be able to take advantage of SMP and
clustering as it becomes supported by the engine.
* A crashing application doesn't kill the engine.
* Developing an application becomes a matter of building a
nice user interface, rather than implementing yet another
plug-in host, solving the multi-file disk streaming
problems all over again and so on... (You *can* still do
that if you like or need to!)
That's what I could think of right now... There are a few disadvantages as
well, but most of them become non-issues with decent plug-in and client/server
APIs. Frankly, I can't think of any obvious problem right now... Off-line
processing? Engine service. Just pipe that data through the engine, and it will
run your net in a non-real time thread. You want to do some processing right in
your application code? Fine, throw in virtual insert jacks in the processing
net, and just stream through your application. Trivial thing to implement in
the client/server API, and it works accross networks as well, if you can take
the latency. (Oh, the engine will measure and/or control the exact latency for
you.)
Tell me what I missed, and I'll solve it! ;-)
And so for this buffer size issue...
Ok, drop frame, samples or whatever from the process() call. Use an event to
specify the size of input and output buffers.
Plug-ins should get the first buffer sizes before the first call to process(),
through a new invention of mine, init(). init() basically does everything that
process() does, except for processing any data (it doesn't get any buffer
pointers). It should be called in non-real time context, so it can safely
allocate memory and take as long to execute as it needs.
The similarity? It handles events. Takes events, sets things up and tells
the engine about any non-standard stuff by returning events. (I intend to have
an event addressing system that allows replying to events. It will be needed
for controllers as well, as you'll want to be able to read them.)
The "buffer size" event should have a channel argument, so that you can set the
size for buffers individually, if the plug-in allows it. Most audio plug-ins
would just say "Homogenous buffer size, please!" (defualt), and accept only
CHANNEL_ALL as the channel argument.
And this system should apply to buffer formats, sample rates and so on...
Everything without lots of binary compatibility breaking stuff in the plug-in
class or instance structs, or even requiring the lazy plug-in hacker to know
about it.
This means that a plug-in can tell the engine wether it supports changing
buffer size/format(weird!)/sample rate/... during operation or not, it can
specify the relations between inputs and output, support mixed buffer size/...
and so on. If you want the default, everything will look like strummin, except
that the properties goes away from the struct.
Oh, events... Quick outline:
#define EV_MAXARGS 2 /* What do we need, really? */
/* For efficient decoding, events are grouped as
* system, control, gui, custom etc... (Custom is
* a group that's reserved for private talk between
* GUI and process() code, and should *not* be used
* for anything that could make sense to use standard
* events for.)
*
* The EV_FLAGS field is used for telling requsts
* from replies and that kind of things.
*
* EV_KIND is the actual event code. Event kinds
* should be declared as sequential enums for good
* optimization posibilities in switch() decoding.
* (That's why there's a CLASS field.)
*/
typedef undigned long event_code_t;
#define EV_CLASS_MASK 0xff000000 /* TIMING, SYSTEM, CONTROL,... */
#define EV_FLAGS_MASK 0x00ff0000 /* EVENT, REQUEST, REPLY... */
#define EV_KIND_MASK 0x0000ffff /* Exactly what event is this? */
/* This is for addressing events, and for knowing
* who to reply to, if the event is a request.
*
* Not sure yet, but I think addressed should be
* a system resource. That is, you allocate as
* many addresses as you need, and tell the engine
* where they are for in some way. So, they are
* basically handlers (opaque data type).
*/
typedef undigned long event_address_t;
/* Timestamps in the event? Nope: EV_CLASS_TIMING.
*/
typedef struct {
event_code_t code; /* What is this? */
event_address_t address; /* Who is this from/to? */
int arg[EV_MAXARGS];
} event_t;
So, timing is done by sending an EV_CLASS_TIMING_TIME (or whatever) event,
specifying how many subsequent events will be affected, followed by those
events. Events that are not covered by a _TIME event are out-of-band, and will
bypass the event scheduler WRT scheduled time. (That is, if the event goes to a
MIDI device, it'll be sent ASAP, as opposed to a timed event, which will be
sent as close to the scheduled time as possible.)
Note: If you're in a real hurry, sending out-of-band events using an engine
function call could make sense. (Otherwise you'll just nicely have to finish
your processing and wait until the engine sees your out-of-band events in the
event ouput buffer you return.)
Events can be handled by some nice macros at the plug-in source level... I
think that would be a nice way to make it all look even simple, and to avoid
some bugs. For example, timed events could be handled something like this:
start_timed_event_block(<event_code>, <event_address>, <time>);
event_t *evp;
evp = new_event(<event_code>, <event_address>);
evp->arg[0] = <...>;
evp->arg[1] = <...>;
evp = new_event(<event_code>, <event_address>);
evp->arg[0] = <...>;
evp->arg[1] = <...>;
...
end_timed_event_block();
Of course, the low level stuff of the event system should *not* change, since
that would break binary compatibility!
Comment? Will men with white clothes come to help me soon? ;-)
//David
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:25:53 EST