Subject: [linux-audio-dev] plug-in design space
From: Eli Brandt (eli_AT_v.gp.cs.cmu.edu)
Date: pe elo 27 1999 - 16:50:28 EDT
General methodological question: is anybody broadly familiar with
existing plugin APIs, and can say what's good or bad about them?
(I could tell you about DirectShow, but you'd just have nightmares.)
Will a new API be enough better that people will adopt it widely?
How about a couple of larger-scale design questions before we get down
to bits and bytes? These decisions affect the complexity of the
plug-in architecture by at least an order of magnitude, based on two
audio-processing systems I've worked on. Oh, make that two orders of
magnitude -- had momentarily suppressed the memory of DirectShow.
* Is this for real-time streaming, non-real-time editing, or both?
I vote a big "both". RT streaming pretty much requires the use of
data-pull, walking backwards up the graph:
> The algorithm is: "I need X samples. How many input samples do you need
> for each channel?"
But the answer may be "I dunno, I'll have to see what the values of my
inputs are." A voltage-controlled time-warper, for example. This
works fine in non-RT, where it's more natural to do data-push down the
graph. Support this?
A follow-up question:
* If non-RT, can output keep running after input has been used up?
And I don't mean just like for the time-warper, where the output may
be longer than the input, but depends directly on it all the way along.
I mean like an echo effect would want to keep generating output until
the echo has dropped to a low enough level. Or do we just say that
it's the user's responsibility to guess how long the echo tail will
be, and pad the input with zeros? I have no opinion on this one.
* Block-synchronous computation?
If all blocks of data cover the same span of time, then each component
gets a complete block at each input and puts a complete block on each
output. This is pleasant. But I'm afraid we're going to be compelled
to leave this Eden.
* What about sampling rates?
Again, life is much easier if there's only one sampling rate. This
loses efficiency, because some of your signals don't need to run that
fast -- in synthesis applications, a factor of two or three.
It also loses generality (the loss that'll hurt more five years from
now) because sometimes you need a higher rate. Upsample - process -
downsample to make EQ work right on the high end, or for distortion
without aliasing. Or you could, by agreement between components,
flatten overlapped spectral frames into a 4x-rate stream, if you have
no better way to pass this data.
I've seen an arbitrary-sample-rate architecture (Nyquist), and it's
not pretty. Possible compromise: an audio sampling rate, and support
multiples and submultiples of it.
* Support multiple sound cards that aren't sample-synchronized?
Without doing full-on windowed-sinc resampling all over the place,
we're not going to be able to mingle data from multiple sound cards.
So be it. But what about events, parameter updates? It's simplest if
they're timestamped in terms of sample frames. Do we care enough
about unsynchronized cards to timestamp events with some global clock
and then deal with conversion? I, personally, do not, but I do wish
Linux could get more drivers for multi-channel cards.
* What's static and what's dynamic?
It would be nice to support variable numbers of inputs and outputs: An
n-to-m matrix mixer, for example, where n and m are determined upon
instantiation of the prototype. Here n and m are probably not
"events" like you'd send to a component-instance. Interesting
startup-ordering issues. And do component-prototypes have the
opportunity to specialize the execution loop of each instance?
* How are feedback loops handled?
If there are no cycles, the flow graph is a DAG and you can just
topologically sort it up-front, or traverse it on demand. With
cycles, you need to be a bit more clever. And do you enforce that
loops not be delay-free? If not, do you let the result vary with
block size, or force block size equal to one sample, or what? I've
never worked with a system that allowed loops.
* Is each `wire' mono?
Here's where I stepped in. My leaning is towards mono wires for
simplicity. Performance is not clear to me. Better memory-access
patterns for interleaved channels. Worse inner loop, the general loop
anyway: "for (int channel = 0; channel < nchannels; ++channel) foo"
where nchannels commonly turns out to be 1 each time.
--
Eli Brandt | eli+@cs.cmu.edu | http://www.cs.cmu.edu/~eli/
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:25:53 EST