Subject: Re: [linux-audio-dev] XAP spec - early scribbles
From: David Olofson (david_AT_olofson.net)
Date: Tue Mar 04 2003 - 18:06:00 EET
On Tuesday 04 March 2003 10.20, torbenh_AT_gmx.de wrote:
[...]
> > Exactly - and that is why you must design in a way that allows RT
> > hosts and plugins to operate without non RT safe actions.
>
> yes and because a nonRT system can do more things it is a superset
> of RT. (but this is just being picky :)
No. If 'B' is a superset of 'A', 'B' must be able to do *everything*
that 'A' does. A non-RT system is (obviously!) not RT, and therefore,
is in no way a superset of an RT system. It is a different kind of
system.
This might sound picky, but the difference is of the "make or break"
magnitude. Consider controlling a heavy industrial robot arm, and you
get the idea.
[...]
> > Well, it's not really an error to implement a XAP plugin that
> > won't run in real time and/or is nondeterministic. However, I
> > don't quite see what you could do in an RT capable *host* that
> > inherently cannot run in RT. Multi-pass subnets with conditional
> > loops...? Can you give some examples?
> >
> > I realize that non-linear time processing falls into this
> > category, but I don't see how that is relevant to a real time
> > plugin API. I can see how it would be *possible* to support it in
> > a plugin API that also does real time processing, but I think
> > designing something like that is asking for serious trouble.
> > There are just too many conflicts between these two ways of
> > working.
>
> yeah i see your point...
>
> how about a flag indicating "dont call my process() from the
> realtime thread, but from another thread"
I've considered that, but dropped the idea intirely. There are two
major problems with it:
1) Plugin and host interfaces are not thread safe.
2) Ripping a plugin out of the net means the net cannot
function properly while the plugin is away.
Of course, 2) is better than blocking the RT thread, but it's only a
minor improvement. Allowing plugins to deal with it allows them to
actually keep processing while waiting for the background work to
finish.
For example, a delay could keep running with the old buffer until a
new buffer is ready for use, and then send the old buffer off for
destruction.
> the event sending in this case needs to be different but this could
> be a different function/macro which needs to be called in this
> case.
That is, an alternative API. What's the point? If the plugin is
running in another thread, API semantics change drastically anyway,
so it just can't work in anything like the normal way.
> note that in galan i have the opengl rendering components which
> receive events also. I would be nice if this could be handled by
> XAP...
Sure. My idea is to do exactly this with GUIs. GUIs would essentially
be XAP plugins, running as clients (JACK style) instead of in-process
plugins. Hosts would have soft/hard RT gateways for this, and events
would be transferred via shared memory, pipes, sockets or whatever
fits the environment and the task at hand. Plugins would never touch
the details, but rather just use the normal event management inlines,
and in the case of the GUI "clients", a small library that acts
somewhat like a simple single plugin host.
[...]
> > Right - but then, why mess with this on the API level, and why
> > use a delay at all? Well, I can see the jitter reduction
> > advantage, but I don't think it's worth it. Just design plugins
> > so that you can tell them what you're going to do first, and then
> > have guaranteed RT response.
>
> how is this guaranteed ?
This is what RT programming is all about; determinism. Don't do
something you can't reliably finish within a know amount of time.
Don't access file systems, don't risk hitting the swap through
malloc(), and that sort of stuff. Really rather simple in theory.
> when i tell a plugin it should load sample X
> it might take hours to download this url via modem.....
Yes - that's why you should *never* assume anything about the time
such actions take to finish. Load your samples before you start you
gig, period.
> it seems like you expect the user to know that and dont
> use the sample until its downloaded.
Exactly. Unless you know a way of working around the laws of nature,
there's no other way to do it. Indeed, the user can be informed that
the sampler is not yet ready, or even prevented from doing anything
at all until it is (the "READY" output) - but that doesn't really
change anything; it's just a nicer interface in some situations.
> the evtdelay and an evtgate are some of the plugins who
> could handle delayed events and i dont see the reasons
> to rule these out.
Sure you can have them, and they might actually be useful in some
situations. (Delays definitely are.) However, they will not set up a
time portal for you, and thus, I don't see their relevance to non-RT
actions. If it's not real time, it's not real time, and there's no
way you can ever make it real time without replacing the offending
parts. (Play directly from flash memory instead of loading from a
hard drive, or whatever.)
> the plugin must do its own buffer splitting, process ramps etc...
> why dont you want it to do
>
> // This if() needs to be a nice macro for fixing wrapping
> // or #define SAMPLETIME gint64
>
> if( event->timestamp < current_time )
> drop_event_if_i_dont_like_it()
>
> even late ramp events could be fixed... if the host did
> not destroy its timestamp by setting it to current_time...
If the host changes timestamps, it will also have to adjust ramp
events. I see no reason why plugins should be forced to consider
this, as they can't really do anything more useful about it than the
host could - and they'll *never* have to do it when they get input
from a hard real time source anyway.
> what are your reasons for this ?
> does it have to do with timestamp wrapping ?
> this could be fixed trivially...
My real reasons have very little to do with implementation aspects,
actually. I'm a control engineer, and the term "hard real time" has
an absolute meaning to me. Late events delivered to real time plugins
make very little sense in my world. If they're from within the net,
they simply won't be late. If they're from the outside soft RT world,
they can be late, but then they should be adjusted appropriately by
the *host*, which will have to deal with the rest of the hard/soft RT
interfacing issues anyway.
> > For example, the delay could have a "max delay time" control for
> > buffer allocation, and a "delay time" control for setting the
> > actual delay. That way, you can use the same plugin for ms delays
> > as well as delays of several seconds, without hard-coding the
> > plugin to some arbitrary huge buffer size.
>
> max delay time is not realtime safe...
Exactly.
> how shall such an event be handled ?
Send it before you start your performance, and wait for the plugin to
get ready. That's how you do it with h/w samplers, some reverbs and
even synths that can't respond instantly. I can't, even in my wildest
imagination, think of any other totally safe way of doing it. Assume
anything, and you're taking a risk.
[...]
> > Sure, but timestamps wrap, and as a results, they wrap into the
> > future if they arrive late. This can be "fixed", but it requires
> > that we agree on a specific maximum allowed time an event may be
> > late.
>
> no problem MAX_TIMESTAMP - current_blocksize
> future events are not delivered to the plugins.
Unfortunately, it's not quite that simple, since events are sent
directly; not through the host.
The obvious example would be a smaller block subnet running inside
another net. The subnet plugins would receive events for the duration
of a big block every now and then, and thus, would frequently see
events past the end of *their* current block. They should never touch
these, but they have to be aware of the fact that the event queue
doesn't necessarily end with a NULL next pointer. (Don't worry about
the "extra" check; it coincides with the audio block size check
anyway, and is only evaluated once per event "group"; ie when you're
about to enter the DSP inner loop after processing one or more events
with the same timestamp.)
> > More importantly though; it's not possible to handle control
> > ramping properly, unless timestamps are respected as part of the
> > data. "Process ASAP" results in quantization, and that will screw
> > up chains of ramps. (XAP ramps are just "aim point" commands, and
> > nothing is guaranteed after an aim point is passed. Plugins are
> > not required to stop ramping automatically, but can expect to
> > receive a new event in time.)
>
> yes... normal plugins can drop this late events or try to
> interpolate to the current time.
>
> there are not much plugins capable of handling late events
> but a delay could handle them.
Sure, but only if they're not *too* late. It might be an interesting
special case, but I don't think it's worth the extra logic,
especially not if it has to be in all other plugins as well.
> > > plugins could report that they received a late event
> > > to the host so it could notify the user or abort...
> >
> > That's just moving a host implemantation issue into plugins.
> > Allowing late events is almost like allowing late audio buffers.
> > The only difference is that with events, it *looks* simple,
> > whereas with audio, it's obvious that you'd need a different
> > protocol. When you look closer at it, specifically at ramping, it
> > becomes obvious that there isn't all that much of a difference.
> >
> > I don't think plugins can do anything useful about late events
> > that hosts can't fix by adjusting the timestamps, and as this
> > never happens within a real time net in a single thread, it
> > doesn't even have to be considered in most hosts. Only hosts that
> > support soft RT connections between multiple threads or processes
> > will need to deal with this, and they can do it inside the event
> > gateways that are required anyway.
>
> yes this is correct.
> but if the plugin could decide by itself what to do with a late
> event everything would be fine.
Yeah, but that would have to be an optional feature. A plugin would
say "you may leave the late event handling part of soft/hard RT
gatewaying to me."
This would require that the maximum allowed delta time between two
events is lowered to less than the full range of the timestamp type,
but that's not much of an issue with 32 bit timestamps. (It's
slightly worse with Audiality, which uses 16 bit timestamps, but it's
no major problem.)
I'm not sure it's as easy as that, all things considered, but I agree
it would be a nice feature. If nothing else, it would allow most part
of the soft/hard RT gateways to be implemented as normal plugins. One
might implement "time smoothing" or something, as an alternative to
just hard quantizing any late events, leaving others where they are.
> > > but why do you want to rule out late events ?
> > > they can be handled in a sensible way.
> >
> > I don't think they can be handled trivially, and I don't see any
> > reason why the API or plugins should consider it at all. Whenever
> > a host makes a connection that can result in late events, it has
> > to make sure the events are adjusted as needed, so plugins get
> > delayed or quantized events.
>
> yes... and how is this decision made ?
If the events come from another thread, and you're not in a hard RT
system, events may arrive late. Anything "further away" than that,
and events most probably will arrive late every now and then.
In short, if there's any chance at all that events arrive late, you
need a soft->hard RT gateway.
> in my proposal the user would wire a delay or a quantizer into the
> net and everything would be fine.
Users should never have to do anything like that. 99.99% of users
would just get a big cartoon question mark above their heads if you
suggest this. :-)
> in your proposal you would have some logic in the host decide what
> to do generating complexity in the host as this question must reach
> the user somehow....
It's implicit, and the user can't do anything about it anyway. Why
bother the user?
As to complexity in the host; rather that than doing it in every
single plugin... It has to be done somewhere either way, unless
you're entire system is hard RT and using locked time bases for
timestamps.
> > > i think this is similar to the soft mode of jack....
> >
> > Maybe, but audio is not structured data, and thus, isn't as
> > sensitive to glitches. You don't have to do anything special to
> > "fix" drop-outs in audio streams without totally screwing up the
> > data.
>
> yes...
> but if the system was robust enough to handle an event dropout it
> would be ok.
What do you do about late events? Travel back in time and do what you
should have done? ;-)
Seriously, I don't think there's much point in doing anything beyond
processing late events ASAP, one way or another. If you get late
events all the time, and they're late enough that you can't play, you
need to fix your setup. No tricks in the world can avoid that. If
increasing the latency to tens or hundreds of ms works for you, fine;
tell the host to bump incoming soft RT events that much. It's just
like setting a larger audio buffer size when you're getting
drop-outs.
> also the user could guard himself against the dropout
> case...
>
> with an eventgate of some sort.
That's just not possible, unless there's a fully hard RT path from the
source to the plugin. Adding a fixed latency can reduce jitter, but
it can't entirely prevent missed deadlines ("drop-outs"), unless the
events come from and are delivered by hard RT subsystems.
[...]
> yes... i see the advantages of running once per block...
> the could be also done by the host compiling a graph into
> an XAP plugin ... i am fine with this method as i see a way
> out of the misery for me :)
Well, here's another weird suggestion for ya': Process zero frames.
:-) For various reasons, this should be legal, and it means "process
all events for the current frame and then return". This would allow
zero latency feedback for events - provided the host can figure out
when to stop looping! *hehe*
[...]
> ok... i just like the idea of galan beeing turing complete :)
Well, XAP isn't a programming language, so that's not a major goal for
us. :-)
> well i dont really know what i want to achieve with this.
That's a bad sign... ;-)
> i have not yet spent a day experimenting with complex
> event logic...
>
> i spend too much time on thinking about our current discussions...
Tell me about it... Countless are the hours we've spent around this
list, figuring all this stuff out.
[...]
> how do you know its ready ?
It sends a CONTROL event setting whatever you connected to it's
"READY" output control to 1. (Non-techie version: It says "1" on it's
"READY" output.) Hook all READY outputs up to a LED on the transport
panel or whatever. You could even have the PLAY button wait for all
plugins to get ready before actually starting to play.
[...]
> > This is not possible, and IMNSHO it's completely pointless to
> > even consider this on the API level. There is *no upper limit* to
> > the time it may take to allocate some memory in a general purpose
> > OS, such as Linux, Mac OS X or Windows. Deal with it, or switch
> > to a complete RTOS with virtual memory disabled.
>
> if the user knows there are still 512 MB free there is an upper
> bound.
I wish it was that simple... Seems like most platforms manage to do
silly stuff even if there's supposed to be lots of free physical RAM.
It's a result of memory managers being optimized for average speed
rather than deterministic execution times. You have to use your own
memory manager with a preallocated, locked pool, or you never know
when malloc() or free() will bite you.
[...]
> ah.. there are soft RT and real RT controls ?
> is that a hint on the control ?
Yeah, that's the idea. All it means is "this control may not respond
instantly." The plugin will still function while implementing changes
of these controls (as in the delay example), and the host or other
plugins don't have to worry about it. Plugins are free to implement
these controls in any way they like, as long as they don't block the
RT thread. Dividing heavy computation over multiple blocks is one
way. Throwing a worker callback is another.
[...late events vs soft RT controls...]
> hmm... i see your point somehow...
> but i am still not convinced...
Well, some of the above might explain what I'm thinking. Here's a
small comparison, based on what I can think of right now:
Late events:
* Caused outside the receiving plugin.
* Can be hard RT, but just not in sync with the net.
* The host can deal with them.
* Plugins don't have to be aware of their existence.
Soft RT controls:
* Caused by the receiving plugin itself.
* May be deterministic, but won't respond instantly.
* Cannot effectively be handled by the host.
* Only plugins that have them need to deal with them.
[...]
> > I'm not talking about constructors, but about loading presets and
> > changing controls. That's nondeterministic stuff, so if you want
> > to be sure you won't screw up half your intro, you simply have to
> > wait for it to finish before you start playing.
>
> yes ... correct...
> but how about streaming components ?
Streaming is a matter of Quality of Service guarantees, which are
closely related to real time. In short, if you want absolute
guarantees for streaming, you need a guaranteed bandwidth quota on a
per time slice basis. For example, "10 Mbps" is worth nothing in hard
RT terms, while "512 kbps average; guaranteed 16 kbit every 100 ms"
allows you to set up sufficient buffering and do rock solid
streaming.
> the ogg_ra component has an input event triggering the loading
> of the next buffer. in galan 0.3.x there is a worker thread
> exchanging 2 buffers with the realtime thread to make this
> operation realtime safe. but this does not handle skipping
> elegantly....
[...]
To put it simple, the only thing you can do is move the problem
around, and do the best you can when you have no input. There are no
magic tricks.
Delay isn't really a solution; it just increases your tolerance. With
QoS guarantees, you can calculate the exact delay/buffering you need
for solid playback, but without it, you'll have to assume that data
will arrive too late every now and then.
As to your current implementation, it's just a matter of granularity.
If you want the RT thread to get incomplete "frames", you have to
have the streaming and RT parts interact more than once per full
frame. Two buffers might be too few. Or you simply don't have enough
buffering. (We're talking *seconds* if it has anything to do with
disk access.)
[...]
> > > ah... the execution path of an event can enter the realtime
> > > thread, and leave it again into another thread...
> >
> > Not quite, since the event itself never gets to the worker
> > thread. The event is received by the plugin, which sends the
> > *job* off to a worker thread. When the worker thread is done, the
> > host sends a "result" event to the plugin, which can then go on
> > with any further, RT safe operations.
>
> but then the delay must be part of the plugin...
There *is* no delay. The plugin just waits until the job is done. No
one knows how long this will take, so the term "delay" is really
rather irrelevant.
> and i think that this is not necessary...
>
> i think the mesh would be easier to understand if
> the delay was an explicit element of the net and not hidden
> in some component.
An explicit element that makes you think you have a hard RT net, while
in fact, you do not? No thanks.
The "delay" is not hidden, but *unknown* and impossible to calculate.
> > > if the realtime thread advanced to much then
> > > the event would have a timestamp in the past.
> >
> > No, becaues there would be no events at all outsid the RT thread.
> > The "result" event is generated by the host as soon as it finds
> > out that the worker thread has finished.
>
> hmmm... what about the opengl thread ?
> i need a way to have different threads and dont want to process all
> events in the realtime thread which is obvious, isnt it.
That's a different story entirely. Another thread, another host...
Soft/hard RT gateways in between, if plugins in it are to interact
with plugins in the audio thread.
> This also a requirement for the graph sorter...
> how do we model this ?
Different nets with soft RT connections between them. If you really
want full blown event processing in the OpenGL thread, I don't think
it makes sense to "sneak" the OpenGL code in as worker threads
belonging to what appears to be audio plugins in the audio thread.
They're better off as separate plugins, with their host(s) dealing
with communication implementation details.
> > > events with timestamps < gen_get_sample_time() are always
> > > processed first: better late than never.
> >
> > Well, you could say the result events are handled like this, but
> > it's done by the host (by picking a suitable timestamp), and the
> > events aren't really late. They just tell the plugin that it's
> > worker thread has finished.
>
> now the host destroys a timestamp which could be fixed by a delay.
What do you want to fix? When you get this event, you will already
have generated one or more blocks of output, and there is *nothing*
you can do about it. Those buffers are probably already on the way
from the speakers to the listener's ears.
The event just tells you that your worker callback has returned, so
you can apply the data it generated, or whatever. What use is it to
know the exact time (in audio frames) the callback happened to
return?
> i dont want delays to be special components... the simply arent.
Right; delays are just delays - but we're talking about asynchronous
processing inside plugins here.
[...building Audiality 0.1.0...]
> >
> > You have ALSA 0.9.x, but this version of Audiality only supports
> > 0.5.x. It's just for raw MIDI anyway, so you might as well use
> > OSS emulation. './configure --without-alsa'.
>
> ok... it compiled... what do i do now ?
> need to look more into it...
Not much you *can* do with that version, without some coding, at
least...
> any exmaples for it ?
There is a simple demo program that uses it to play a MIDI file with
custom sounds, but that's about it. Enabling RT MIDI input is just a
matter of adding a flag to the *_start() call, but there's only OSS
rawmidi input in 0.1.0; no ALSA sequencer.
0.1.1 might be a bit more interesting, but I have this habit of
thinking the "top priority" part of the TODO is too big as soon as
it's non empty. ;-)
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
This archive was generated by hypermail 2b28 : Tue Mar 04 2003 - 18:45:53 EET