Subject: Re: [linux-audio-dev] Re: Plug-in API progress?
From: David Olofson (audiality_AT_swipnet.se)
Date: ma syys 27 1999 - 17:01:33 EDT
On Mon, 27 Sep 1999, Paul Barton-Davis wrote:
> >An RTL driver can easily catch events and time them within 5 usecs or so on an
> >average Celeron box. The hardware can be more troublesome...
>
> regular Linux can do the same thing - remember, RTL doesn't improve
> the latency of an ISR (sti/cli excepted).
^^^^^^^^^^^^^^^^
That's *exactly* the point with RTL.
As an RT/control kind of guy, I'm not very interested in the average case...
But yes, if the jitter of standard Linux IRQs isn't a problem, you're right.
(IRQs are not preemptive on x86 hardware, unless you make them so by
reenabling the interrupts in the start of the handlers...)
> >> thats still a lot
> >> of samples that may or may not have been generated so far by any given
> >> plugin. The timestamp will be accurate, but there is no way to sync it
> >> to the sample generation process.
> >
> >If you know the offset between the time base of the event and the time base of
> >the output device, there's no problem.
>
> sure, but there is no fixed offset if its measured with sample
> resolution. you simply don't know the difference at any particular
> point in time.
Why not?
RDTSC gives you the absolute time right now, and an IRQ from a sound card tells
you that the card just reached a known state - usually that it has
played/recorded the requested number of fragments, and is going to need a new
buffer after the current fragment has been played/filled. The sample frequency
can be considered as known to be what you requested, or you can measure the
exact rate compared to the CPU clock (if you're using RDTC) if you really don't
trust the sound card. (Pointless, as you'll never see an error bigger than a
fraction of a % of a sample even with a rather crappy card, with normal real
time audio buffer sizes.)
Now, if you get the current time at *any* time, you can just check the latest
buffer timestamp from the driver, and calculate what sample is playing right
now. (Or calculate whatever you want to know, like the offset between two
cards - in samples or whatever...)
> >> The best you can do is sync it to
> >> the state of the DAC, as understood by the engine, which is well known
> >> immediately after a write to the DAC has returned, but not known at
> >> any other time.
> >
> >True. With OSS and ALSA (currently, at least). What about an IOCTL that gives
> >you the exact time of the next sample you'll get if you do a read()?
>
> ALSA provides an ioctl that does something fairly close to this, so it
> wouldn't be hard to fix i think. but this still won't give you sample
> accuracy unless you read the data from the driver in single samples.
>
> Consider the following scenario:
>
> * engine has just finished a control cycle, and has
> returned from sending data to the output interface (DAC,
> ADAT, whatever). We assume that it blocks until space
> is available there, assuring us that we are synced with
> the output stream.
>
> * engine grabs a control-cycle's worth of input samples
> to make them available for the next cycle of plugin
> execution.
>
> * engine starts plugin cycle.
>
> * interrupt occurs from some external h/w source (not audio). its
> timestamped, preferably by the driver, but possibly in user
> space, and the event it represents becomes available for
> use by the engine.
>
> * the engine finishes the plugin cycle, and sends the data
> to the output interface.
>
> * the engine gets the cycle's worth of input data.
>
> * it starts the next plugin cycle, queueing the event in the
> plugins' event buffers.
>
> OK, so how is that event supposed to have sample accurate sound ? All
> we know at that point that the event occurs is that a control cycle
> has started.
Why are you forgetting that you can calculate the exact play time of every
single sample of the buffer you're about to generate? The sound driver just
told you when the first sample of the next buffer will be played (or similar
information), right? (I'm assume we're dealing with *real* multimedia drivers
and an API to handle them, of course. ALSA will probably do this, if it doesn't
already.)
> We have no idea how far along we are, because we could be
> in the middle of any given plugin, or the beginning or the end, or in
> between them. We could estimate how long the cycle will take, based on
> previous cycles with the same net, and then judge from that, but this
> is not going to provide sample accuracy.
With a timestamped event system, it doesn't matter where in the cycle you are.
It's actually a system that could run off-line just as well, using events from
a sequencer.
> The best I can see is to use the ioctl call to determine the current
> time as far as the *output* interface is concerned. You can't use the
> *input* interface, because it will give a time based on the samples
> you've already read, rounding its accuracy to the buffer size.
Nope. Just ask the input and output cards about the start time (or whatever) of
each buffer, and you have the exact offset between them. That is, the offset
you need to translate the event timing accurately. (You don't have to do this
very frequently BTW, unless the cards have unstable oscillators...)
> But this implies that timestamping happens in user space, because
> otherwise its a cross driver call, which may not be a very good idea
> from an ISR context. If its in user space, I don't think that the
> timing will be better than 0.5ms, and even that is with the user
> thread running SCHED_FIFO, and presumably not the engine thread - bad
> news on a UP box ...
You shouldn't have to deal with this with sync capable drivers. All you need is
a way to get audio data timestamped using a global clock. The CPU clock should
do, I think.
> >> What *are* the events that might tell a plugin to "do something" ?
> >>
> >> The class of events that I imagine to have important time properties
> >> are those that reconfigure the processing net. Turning on a
> >> plugin. Turning it off. Not much else.
> >>
> >> When someone tweaks a MIDI CC, I expect the effect to take place right
> >> *now*. When someone twiddles an on-screen control element, I expect
> >> the effect to take place right *now*. I don't see where a notion of
> >> "when to do things" enters into the area of parameter updates.
> >
> >Automation. Live systems are obviously *very* different from studio systems...
> >:-)
>
> Nope. The automation I've seen in ProTools, Logic, the Pulsar and
> others tells something when to change a parameter. It doesn't tell a
> plugin to *do* anything.
About the same way VST does it (although it uses callbacks to change
parameters)... but not for VST 2.0 soft synths. They use a timed event system
similar to my design.
Anyway, "when to do things"... Simple answer: Do <parameter change> at sample
<sample> in this buffer makes a lot of sense to me, as it means you get
automation resolution that's completely independent of the buffer size. Ruining
things like a compressor side chain gating another track just because you want
to save some CPU by using bigger buffers when mixing down isn't a very goo
idea, it is? And you certainly need small buffers to get decent resolution in
the first place...
> >> I consider a plugin to be a little piece of code that generates
> >> samples according to an algorithm and the state of its parameters. It
> >> doesn't change what it does unless you (1) change the algorithm, which
> >> is nonsensical - the plugin *is* the algorithm (+) or (2) change the
> >> parameters. The parameters can be updated in real time. So I can't see
> >> what kind of effects you might be talking about.
>
> >The audio clip player of a hard disk recording engine. It's actually
> >*required* to work with at least sample accuracy.
>
> thats a different issue. we're talking about events that alter the
> behaviour of the plugin. perhaps i'm missing something here.
audio clip player == sampleplayer
Or; someone will have to make sure the audio clips get mixed in at the exact
sample positions they're supposed to - and I intend to allow plug-ins to do
this kind of work, controlled through the event system.
> >Plug-ins that are modulated from envelope generators in a soft synth
> >or sampleplayer. These need to stay in sync with the audio data they
> >process,
>
> are you talking about envelope generators that are part of the same
> system ? once again, these just do parameter updates - they do not
> change the code that a plugin executes.
What's the difference? You still need to make sure the "update" takes place at
the right position in the buffer - which is impossible to do without some form
of timestamping, at least in the environments we're dealing with.
> >2) An any machine (UP or SMP), this plug-in may execute within a
> > fraction of the time it takes to play the buffer it's changing.
> > That is, the time at which the plug-in recieves the event, and
> > acts upon it has a very unlinear relation to the buffer time.
>
> Right this is a point you've made before. I accept it, and I'm not
> trying to get around it. But my point is that the example shows how
> async parameter updates remove any need for events or timestamps for
> the most common class of events. Whether or not they actually occur
> asynchronously with plugin execution is a side-effect, and mostly
> irrelevant.
I don't think so. How would you make sure an automation system repeats the
*exact* same control sequence that you just recorded? (I think that's the most
obvious scenario for making the difference obvious.)
Oh, well, perhaps the discussion below the code example will make more sense.
>
> [ ... code criticism elided ... ]
>
> OK, OK. I didn't mean this as a piece of production code, just psuedo
> code for pedagogical purposes. If I have to write it reasonably
> efficiently, it would look something like this:
>
> process (MyClosureType *closure)
> {
> unsigned int nframes = control_cycle_frames;
>
> switch (*closure->switch_state) {
> case 1:
> while (--nframes) {
> ... foo ...
> };
> break;
> case 2:
> while (--nframes) {
> ... bar ...
> };
> break;
> .
> .
> .
> }
> }
Yes, that looks a lot better. :-)
However, now, it will ignore asynchrounous changes of switch_state while
processing... Which means that if a change occurs just before the call to
process(), it'll take effect starting with buffer[now], while the external
event casuing the change coming in just a little later, will make it take
effect starting with buffer[now+1].
With timestamped events, the external event will be timestamped, the
timestamp will be converted to the sample time base of the plug-in, and the
process() function will get an event that tells it to perform the change at an
exact place in the buffer. What you get is a chance to replace one cycle of
jitter with one cycle of constant delay. (In the ideal case; add enough delay
to make sure the system's scheduling jitter doesn't cause events to arrive one
buffer too late every now and then.)
> >> But there aren't very many events that do that ... Most
> >> of them are just parameter updates. These don't change what the plugin
> >> "does" in the sense that it needs to be told about it. They may well
> >> change the output of the plugin, but it doesn't have to care about
> >> that :)
>
> >MIDI events? I'd say they change things pretty much in soft
> >synth... (Note: I'm not thinking about soft synths built from lots of
> >small plug-ins here. This is raw, optimized code for generating N
> >voices.)
>
> ah, thats what I mean by having a different idea of what a plugin
> should be.
Yep, I though so!
> i don't want to use such a piece of code - its just doesn't do what i
> want, which is to be modular, flexible, reusable and extensible. ok,
> so you gain a few percent speed by coding it that way, but by next
> year, we'll be back to square one on that count as long as N stays the
> same (which I'd argue it basically does).
>
> every time i want to rip out the oscillator algorithm, it means
> recompiling the plugin. every time i decide i want to alter the filter
> design, it means more code, possible slowdowns if I want to be able to
> select different filters at runtime, etc. for me, the whole attraction of
> "plugins" is that they avoid this kind of thing.
very good point...
> so, yes, in the case you describe, MIDI events would be significant,
> since they may fundamentally alter what the plugin does. but in a
> system where all a MIDI note does is to insert a new copy of the
> plugin into the processing net, there is no reason to communicate such
> events to plugins at all.
...but I don't fully agree that it's that black and white. Just because
proprietary soft synths often work like you describe, there's no requirement to
do it that way. (It the event system gets *that* expensive, I'd better scratch
it and start over...) On the contrary, I intend to use very small plug-ins as
synth modules.
The *point* is that, as long as I don't require feed-back with a shorter delay
than the cycle time, there's no need to decrease the buffer size to get
acceptable resolution of the control. One plug-in could route MIDI style events
forward to oscillators, envelope generators, VCOs etc, without limiting the
resolution to the buffer size.
And, just to point out an important difference between our systems; my system
normally doesn't remove and insert plug-ins at all as a response to a real
time events. (The engine will temporarily disable silent plug-ins if they're
smart enough to notify the engine about their state, but that's not quite the
same thing.)
Note that the same rules apply to event buffers as to audio buffers - plug-ins
that aren't sending events to other plug-ins the "wrong way" (ie cause
feedback loops) can send events to other plug-ins that will be called later on
during the cycle. The net builder keeps track of this, just at it does for the
audio data flow.
That means that one plug-ins can send an event to a plug-ins farther down the
processing net, and tell it to change some parameter *at the exact same sample*
as an event it recieved itself.
> [ just a note on something related to this: polyphonic plugins are one
> area of quasimodo where things don't feel quite right. because there
> is only one copy of the output buffer for the plugin, each "voice" (an
> instance of the plugin within the current DSP program/net) has to
> cooperate a little bit to avoid trashing the current contents of the
> buffer. This is easily accomplished: the plugin uses a special DSP
> instruction that resets the output buffer(s) once per control cycle
> regardless of the number of voices, and the plugin uses "+=" instead
> of "=" when assigning values to the output buffer, so that we end up
> with the summed output of all the voices, not just the output of the
> last voice to run.
>
> even so, it bothers me.
> ]
This is one part of signal processing that seems to quite often result in
not-so-sexy solutions... VST does it by using two versions of process(); one
that overwrites (to avoid the clear operation - pays off when you're dealing
with insert effects that only output to the next insert effect in the line),
and one that adds to the output buffer. I'll use a similar system, where the
very inflexible mixing process() call is replaced with process_mix(), that is
required to feature a standard output gain control. That means you can build a
virtual mixer without using extra plug-ins for send levels, gain and so on.
> i don't think that our aims for Audiality and Quasimodo really differ
> at all. my purpose in discussing all this has been to try to see if I
> am not noticing something that would prevent Quasimodo from evolving
> into exactly the kind of system we are both talking about. I mean, as
> far as offline processing goes, Quasimodo already handles Csound
> scores, which in many cases can never be handled in real-time. You
> just turn on Quasimodo's "tape" interface, shutdown the DAC, and let
> it run. By morning, your 512-piece orchestra, which individual reverb
> for each instrument, should have finished playing "Happy Birthday" or
> its equivalent in Sweden :)
Certainly, our aims are very similar. However, I think there is a little
difference in the perspectives. The event system/direct access discussion isn't
much more than a matter of me being very careful about getting the best
possible accuracy at an acceptable cost. (And I'm known to have very high
demands - even if it nearly kills me sometimes...)
There are more important (possible) difference though.
First, I'm not quite happy with "If it sounds good, it *is* good" in this case
- future developments may render systems built after that motto obsolete,
especially in the case with digital systems. (As in: "Analog distortion is
useful, while digital distortion just sucks." Applies to other digital specific
problems as well.)
Second, I don't think separating audio, video and other data formats into
completely different worlds is a good thing in the long run. That's why I want
a powerful, high resolution event system, and support for dynamic buffer sizes.
It's also the reason why I *don't* want audio specific stuff in the low level
details of the API. It should be generic, like a real time round-robin RT
scheduler with IPC functionality optimised for high bandwidths and high
accuracy real time communication. As a system based on cycles and timestamped
events can be run without the time base being real time, the off-line
processing capabilities come as a (virtually) free bonus.
Finally, I'm just about crazy enough to believe this is actually possible to
achieve. ;-)
//David
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:27:12 EST