Re: my take on the "virtual studio" (monolith vs plugins) ... was Re: [linux-audio-dev] ardour, LADSPA, a marriage

New Message Reply About this list Date view Thread view Subject view Author view Other groups

Subject: Re: my take on the "virtual studio" (monolith vs plugins) ... was Re: [linux-audio-dev] ardour, LADSPA, a marriage
From: David Olofson (david_AT_gardena.net)
Date: Sat Nov 25 2000 - 04:09:12 EET


On Sun, 19 Nov 2000, Paul Sladen wrote:
> On Sun, 19 Nov 2000, David Olofson wrote:
>
> > On Sat, 18 Nov 2000, Benno Senoner wrote:
> > > but what we need is a communication system which must be very lightweight
> > > even when making/destroying connections (since we want to place patches in real
> > > time).
> >
> > How about changing a pointer or two per connection, possibly using a
> > very light weight function call? LADSPA does it that way, and MAIA
> > (MuCoS) won't make it heavier. (It might even eliminate the function
> > call if you can settle for connecting in between buffers. There might
> > be a good reason to have that kind of notification for the plugin,
> > though.)
>
> By my reckoning there are 3 types of parameters that are going to get
> passed to plugins:
>
> * State:
> These are so fundemental that the plugin needs a "power cycle" (an
> "end()", lollowed by "set(parameter)" calls, followed by an "end()")
> changing the order of a parametic equation requires re-evaluation to
> produce new co-efficants for example.
>
> * Syncronous:
> Must be updated "as part of the RT chain"...
>
> * Asyncronous:
> Can be updated anytime, but you've got no guranttee that the new
> values are going to be used till the next pass.
>
> All three are updateable from outside the begin()..end() pair via a
> function call. (the call to begin effectiveally makes a move from non-RT
> to RT). Only Sync and Async variables can be updated while in "RT" mode,
> and Sync variables only via a function call inline within the plugin
> call.. Type 3, asyncronous parameters can just be updated via a plain old
> "poke" into memory, so we would need some way to publish the memory
> location (relative to a particular instance of a plugin).
>
> Therefore async variables are much better from the POV of being updated
> from a non-RT server process.

Yes, but they're hopeless on anything but UP and SMP systems.
Network transparency isn't possible without hardware shared memory,
unless some kind of protocol is used to notify the host about changes.

Whether or not this applies to connections is another matter, and I'm
not entirely sure if it's safe to assume that no function calls or
events will be needed. Connections are totally local in LADSPA, VST,
MAIA etc, as it's just about telling the plugin where a shared buffer
is. Hopefully, there's nothing more to it than that... as long as
*plugins* don't start moving the buffer pointers of other plugins
directly. (If the host does it, the host gets to be aware of
networking. If some plugins do it, we're in trouble.)

> > > Plus for low latency apps, we need lock-free communication between the
> > > RT and non-RT parts of the app.
> >
> > Pointers to linked lists of events, passed via lock-free FIFOs. (No
> > lock-free operations on the single event level, that is, thus keeping
> > that outside the plugin API.)
>
> The ideal is asyncronous variables that can be updated RT, once per lopp
> simply through a list of "address", "datum" pairs.

Once per loop is not accurate enough for all applications, and makes
the output depend on the current engine settings. (The buffer size is
*not* to be considered a timing resolution setting - that's a big
design mistake that make many earlier designs extremely inefficient
in some situations.)

As to the rest, that's basically what the MAIA Property Protocol is
doing. There are "commands" to read and write single properties and
arrays of properties, and to request properties to be sent. However,
the "commands" are timestamped at sample accuracy (as all other
Events), and there are Channels and Indexes, rather than just
addresses. (Details may still change.)

However, this is not enough. You can't efficiently and reliably
implement true Commands (like MIDI Note-On/Off) on top of such a
protocol, as that would require either

        1) that plugins check all addresses to see if they're
           variables or Commands, or

        2) that plugins check all command variables after every
           variable change.

1) is inefficient and messy and 2) is obviously out of the question
for any serious work.

As of now, MAIA solves that on an already existing and unavoidable
level: the Event decoding. Every event has an "opcode" that tells the
plugin what to do, and the rest depends on what the opcode is. Some
events need no additional decoding, while others may require
additional levels of switch()es to decode. The design policy is to
keep heavily used protocols direct and efficient (minimal decoding,
no chains of special cases etc), while "lower frequency" protocols
can be more high level.

> > > For example a slider GUI element want to display the current volume value of
> > > an audio track, which gets automation data from another source.
> > >
> > > In this case I want that the GUI slider gets all new volume values from the
> > > audio app, to reflect the "motorized-fader" effect.
>
> Perhaps a list of "address", "message name" pairs so that the RT process
> could grab each of these values of pack it up and add it to the lock-free
> buffer as a message to be forwared asyncronously to the reciveing [GUI]
> element.

Why invent a special protocol?

I'd just give the GUI element an event port (or even make it a
complete plugin), and send normal Property events to it. (With MAIA,
I'd just have the automation send events to all targets - possibly
by passing the relevant outputs from the automation plugin through
"splitters/filters"; basically Y-splits that "downsample" one of
their outputs not to stress the GUI excessively.)

> > > With a lock-free fifo this would be ideal since even if the GUI element is non
> > > RT, it does not loose any data opposed to the model where you are simply using
> > > a shared variable.
> >
> > Nice chance of implementing some sensible filtering algo, that is.
> >
> >
> > > (Plus the shared variable will not work with datasets > 32bits because of the
> > > atomicity problems)
> >
> > (Not even 32 bits are safe on all archs, as we have seen.)
>
> This solves the problem for single datum out/input, but I don't know how
> it would work with arrays... the output from a Graphic-Eq par exampla.

A sofisticated MAIA style approach would be to pass the data in
shared buffers via the Share Protocol. Simply put, that means that
you have a circle of buffers (in shared memory if possible) which are
managed by passing events between the communicating plugins.

When the sender has a buffer filled in, it passes ownership over to
the receiver and then forgets about the buffer. After the receiver
plugin is done with the buffer, it passes it back to the sender
plugin.

Note: This is the totally generic, network transparent, hard
      RT safe etc etc solution; not exactly optimized for passing
      soft RT graphic EQ per-band level info to a GUI.

> > > The same must work in reverse order (a slider controlling a volume value):
> > > use the same fifo approach.
> > >
> > > If I want network transparency, I simply write a "proxy-element" which
> > > has as input the lock-free fifo and as output a network connection or vice
> > > versa.
> >
>
> > Exactly.
> The key to all this is the asyncronous (non-RT) messageing server that
> runs and communicates, passing all the messages around.

It *can't* be non-RT in an RT system - "asynchronous" just means that
the senders and the receivers aren't running in sync. How hard the
RT is for various implementations is another matter, though...

Now, my reaction here might be because I make no definite estinction
between the GUI protocol and what the plugins use inside the engine
thread. OTOH, a GUI with too high latencies is pretty useless, so it
should actually be pretty hard; ie no more than a few ms worst case
latency. If it's worse than that, the GUI becomes little more than a
cute toy when it comes to RT control. (Well, perhaps that's what
people expect from GUIs these days anyway... The days of 2D action
games are since long forgotten, it seems.)

> > > I admit that this asynchronous communication method is not easy to deal with
> > > (even on the same machine) since receivers need to be woken up or do active
> > > polling.
>
> We just need to provide a LADPA-GUI support library that will contact the
> local messageing server, and block until the reply (hopefully) arrives,
> then reurn to the call with the value requested.

I don't think the average plugin should have a specific GUI
interface. Some plugins *should* have a private language to
communicate with their GUI parts, but standardized protocols are much
more useful, and should be encouraged.

Also, a specific GUI API leaves it to the plugin author to decide
what goes into the GUI, and what is published via the *real* plugin
API. If everything is to be "automation enabled", everything has to
be published through two interfaces, or all hosts have to use two
interfaces to communicate with plugins. That makes very little sense
to me, especially as many here have said that GUIs should be
*optional*, and that plugins should be *fully functional* without
their GUIs, as for as possible.

> > > And in order to avoid zillions of calls to write() within the RT audio thread,
> > > we should delegate the wakeup of receivers to a butler thread which gets woken
> > > up by the RT thread and gets informations about whom to wake up using shared
> > > mem.
> > > That way the RT thread can send data to a huge amount of receivers without
> > > fearing a dropout.
> >
> > Something like that, yes... (A kernel module would be nicer in away,
> > but unless it really provides better performance, it's probably not
> > motivated. Unless of course, we start talking about security on
> > multiuser systems...)
>
> Hum getting vibes of non-elegance if we're talking about kernel modules
> just to implament your messageing API...

This is about more than an API, at least if it's to be used for
anything but GUIs. For soft RT in the 10-30 ms latency range, it
shouldn't matter where it's implemented, though.

The kernel module is only required if we have to ensure hard RT and
peak latencies in the ms range - maybe not even then with some kind
of designs.

> I think the future lies in having as much of the setup/processing done in
> non-RT space as possible, The maximum that the RT stuff should have to do
> is to talk to the sound card, and talk to a pair of lock-free FIFOs.

That doesn't work, unless we're talking about high end audio cards
with on-board DSP power. (In that case, we wouldn't need a plugin API
for native processing plugins anyway.)

The whole point with having hard RT in the OS at all is to do RT
processing, so that we can get serious work done properly without
$,$$$+ dedicated DSP hardware. Buffered processing already works
"fine" on MacOS and Windows.

> > > Handling the sending of data from non-RT elements to RT-elements is easier,
> > > since all the non-RT element has to do is to write to the lockfree-fifo.
> > > The RT-element will poll for new data at every processing iteration.
> > > (in the case of audio each time a fragment is played).
> >
> > Yep.
> >
> > > Now my question are MCOP or libsigc++ capable of such things ?
> > > (I mean: lock-free async communication plus VERY lightweight functions to
> > > to connect/disconnect senders with receivers).
> > >
> > > If these APIs are not suitable for this, then we have either to change them or
> > > write one from scratch.
> > >
> > > David O. how would MAIA compare with MCOP/libsigc++ in that area?
> >
> > I'm not sure I know enough about MCOP or libsigc++ to comment on
> > their implementations, but I think the above covers my version. As to
> > the API, MAIA differs basically in two ways from RPC-like systems:
> >
> > 1) You simply send an event; you don't wait for the
> > destination to get it, deal with it, and return a result.
> >
> > 2) You timestamp events to be able to control exactly when
> > the destination will get the event.
> >
> I'm not sure we need this, the delays we are talking about are stupidly
> small anywhere, and just think abou the delay that the mouse-interuppt
> handler is going to cause operating at "only" 1200baud.. I this particular
> moment can't think of anything that needs more than just "update this as
> fast as you can if you could... and keep me posted". I'm obviously not
> thinking about the right things.. if someone could give me an example to
> enable me to start thinking around this one.

Well, the confusion is because we're speaking of different things.
The MAIA event system as a GUI interface is just a bonus - the *real*
point with it is to eliminate buffer splits and function call
overhead at high control rates, and to enable engines running in
different threads, possibly at different sample rates and/or buffer
sizes, or even on different machines to communicate efficiently.

Oh BTW, latencies are not really stupidly small if you're controlling
something over a loaded network. Timpstamped events can help a great
deal in such situations, by guaranteeing that timing jitter doesn't
translate to unsmooth controller movements. Then again, if you're
serious about using a remote control GUI, you can easily avoid that
kind of problems - and avoid the jitter->delay translation effect as
an extra bonus.

> > > Without these characteristics, my "virtual studio" idea cannot be implemented.
> > >
> > > (You know: all "applications" runs as "plugins" of a main host, which
> > > never never never blocks (*) (mark the last 4 words .... ) )
> >
> > The MAIA model assumes that it's ok to send events from plugins as
> > well as from applications outside the engine, or from plugins in
> > other engines on other machines, so no problem there.
>
> Starting to add (perhaps unnessecary?) complexity.

To the hosts. The plugin API doesn't show many signs of the existance
of these possibilities. If you don't want the complexity in your
system at all, just use a simpler host to run your MAIA plugins.

> Think about debugging
> these type of asyncronous multi-threaded realtime. I recall RMS having
> talking bits about GNU Herd. I'm not sure that we want to go quite this
> far... nor have the ten years in which to get it working. just my
> 0.02ukp...

Well, that's all up to people who feel like doing serious work in the
Audio On Clusters domain - I'll just make sure that the plugin API
doesn't kill that path to superiour power by design. :-)

As to any "special" features that would be required to manage latency
and other things in such environments; those are nothing more than
the same features you would expect from a serious audio plugin API.
VST 2.0 has most of them, if not all...

> > > (*) audio I/O is the exception but we all know that this provides a good
> > > determinism. (see low latency tests))
> >
> > Well, that's the time base that drives the host. (*Truly*
> > non-blocking hosts always do nasty things to my machines... ;-)
>
> The only thing that I can see that should ever block is the plugin reading
> from the audio card, and awaiting for the next "packet" to come through.

Yep.

//David

.- M u C o S -------------------------. .- David Olofson --------.
| A Free/Open Source | | Audio Hacker |
| Plugin and Integration Standard | | Linux Advocate |
| for | | Open Source Advocate |
| Professional and Consumer | | Singer |
| Multimedia | | Songwriter |
`-----> http://www.linuxdj.com/mucos -' `---> david_AT_linuxdj.com -'


New Message Reply About this list Date view Thread view Subject view Author view Other groups

This archive was generated by hypermail 2b28 : Sat Nov 25 2000 - 06:54:29 EET