Subject: Re: [linux-audio-dev] Plug-in API design decisions; please consider
From: David Olofson (audiality_AT_swipnet.se)
Date: ke syys 15 1999 - 16:30:41 EDT
On Tue, 07 Sep 1999, Eli Brandt wrote:
> Old message here...
>
> David Olofson wrote:
> > .-------
> > | Are we concentrating fully on audio-only, or
> > | should we take other forms of data into account?
> > `-------
>
> And sound in general is more than just sampled audio! I would love to
> be able to deal with short-time spectra, LPC frames, timestamped audio
> fragments, time-varying matrices, who knows what all. Going down this
> road does make the system unprecendentedly involved....
Again; the event system... And things like time stamped audio fragments
shouldn't be visible to normal plug-ins. They always deal with solid streams,
and things like playing audio clips should be handled by special plug-ins,
talking to a multi channel streamer/caching daemon.
An audio plug-in just processes buffers of audio samples, not structured data.
A MIDI plug-in recieves, reacts on, and transmits events. A video plug-in would
process buffers of one or more frames (look_ahead and skip_behind would still
apply, of course).
It's a matter of drawing the line between "transport layer definition" and
"data format details" - not finding out what should be supported, and then just
throwing it all into the plug-in/engine specification. Or; "Does the UNIX
file system/device driver API understand everything it can handle?" A plug-in
host should host plug-ins, not deal with details of exotic data formats. The
details of a specific data format belong inside plug-ins that
generate/process/convert data of that kind, and inside drivers/system
interface plug-ins, and I can't se many reasons to break this rule.
We're basically dealing with a real time streaming file system. There are
"applications": plug-ins, "drivers": converter/driver/system interface
plug-ins, some of which may be initegrated with the plug-in host, and there is
an "OS": the plug-in host; also know as the engine. What we need is very fast
"task switching" (for small buffer sizes), accurate time stamps, real time
guarantees and some other requirements that normal file systems cannot handle,
but other than that, the way I see the plug-in API is as a kind of interface to
a very specialized OS. However, not as specialized as only being able to handle
data formats natively known by the engine....
> If people who program streaming video think what they'd want in a
> plug-in system is the same as what we'd want for audio, sure, at least
> reserve field-space for merging it in. I'm leery of going out and
> designing a video-processing system myself, since I don't do video and
> wouldn't want to use a video guy's idea of an audio system either.
> (Consider DirectShow, formerly known as ActiveMovie. You can tell.)
I agree. We don't need that kind of mess, and as far as Audiality is concerned,
that much crap in between the engine and the plug-ins would render the API
useless for very low latency applictions.
> Video people have god-awful synchronization issues that I never have to
> think about.
How? Is it trivial to synchronize multiple internal and external audio and
MIDI devices? Well, good looking video frame interpolation (which is usually not
done, BTW - they just drop or repeat frames) is more complicated than
resampling of audio data, but the problems are still very similar. You have
streams of frames that should all be synchronized to a master, and you have to
do something about frame rate differences.
> Their frames are much longer than mine, so they might
> consider it reasonable to hog the CPU for 20 msec.
You get the same problem if you split audio processing over a very low latency
thread for real time synthesizing and monitoring, and a "high" latency thread
for playing and processing stuff from disk or sequencer. This is where a real
RTOS comes in handy.
> I dunno what else
> might come up. You know, the more I think about this the more I think
> the job is quite big enough already.
Yep, but doing things where they should be done (that is, not always in the
engine implementation, or even directly supporting it in the plug-in API) will
actually make things a lot easier for everyone. Engines and plug-ins will be
more reusable, and won't have to care about stuff they won't _actively_ deal
with anyway.
You might have noticed one reason why proprietary standards are designed the
way they are: They force everything to be upgraded in parallel, which in turn
keeps up the cash flow. This is NOT one of _our_ design goals, IMO, so we
shouldn't be to quick to copy the Big Guys. Some things are not what they seem
to be...
> A counterargument is that if we ever want video/audio sync to work, we
> probably have to do the work up-front and get it right now.
Yes. So why not keep things generic up to a reasonable level, so we can turn
an audio engine into an audio/video engine by throwing in some plug-ins and
drivers/interface plug-ins, and possibly optimizing a little by making the
engine natively aware of some new data formats? Even if it requires some extra
thought now, I think it'll pay off in the long run.
//David
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:27:12 EST