Subject: Re: [linux-audio-dev] Thoughts on VST, LADSPA etc
From: David Olofson (david_AT_gardena.net)
Date: Wed Apr 05 2000 - 02:03:23 EEST
On Mon, 03 Apr 2000, Richard W.E. Furse wrote:
> (1) VST
[...]
> (1.2) IMHO, VST has a good plugin API for effects processors and an okay
> plugin API for MIDI synthesisers. But not for unit generators in general.
> The biggest problem I have is the absence of control outputs on plugins -
> it isn't even possible to build a control-rate envelope generator in VST.
> Several other historical features are somewhat unfortunate, particularly
> the rather clunky control input procedure (that will become very ineffic
> ient when fed a high rate control signal) and the [-1, +1] and [0, +1]
> constraints on audio and control data. It's also rather bulky for my
> tastes, probably the result of piecewise development.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Yes, this is what worries me about further extending this API. IMHO,
there's too much overlap already with both the VST 2.0 event system
and the parameter interface.
> (1.3) VST has a large number of plugins that could be compiled for Linux
> without too much trouble. This is an enormous benefit that cannot be
> overstated.
Very true. Still, I wouldn't want to see the restrictions of VST
ported to all new Linux software. Hopefully, it won't be too hard to
port the plugins to a more powerful API. Wrappers can be used until
the new API is generally accepted.
> (1.4) VST has potential licensing issues for source distribution but I
> suspect these could be got around with a little discussion.
Unless this can be resolved, VST is pretty much out of the picture,
IMHO. If we are to write Free plugins for an API, we'd better make
sure we're not running into trouble later on, such as some decision
by Steinberg leaving us unable to use the API. (I don't trust anyone
as long as there's a marketting department involved.)
> (2) LADSPA
>
> (2.1) LADSPA is designed from the ground up as a plugin API for unit
> generators (although explicit use of this terminology was stripped out at
> an early stage). As a result, IMHO it does a better job at handling the
> 'general case' but isn't so good at the specialised cases, specifically the
> synthesiser, because of the absence of event support.
Yes... The problem with this is that it to some extent contradicts
the "unit generator plugin" idea. This API should offer *more*
detailed control (more control data, as in "virtual synthesizer), and
less fancy features, not just less fancy features...
> (2.2) LADSPA2 could be generated by extending LADSPA1 just to include some
> kind of very simple event mechanism, thus providing a logical superset of
> VST SDK2 functionality. I'm not yet sure this is appropriate. Also see (4)
> below.
Possible, but I doubt an event system can be both simple enough and
truly useful. OTOH, an event system is capable of replacing parts of
the current API, which allows you to make up for some of the extra
complexity. This, however, is where it becomes obvious that event
based systems should be different from the ground up to really make
sense.
[...]
> (3) Audio/Event Bus
[...]
> (3.4) The audio/event transmission mechanism does not need to be specified
> in the API itself - a library (and perhaps a hub process/kernel module?)
> could be provided.
This means too much overhead, IMHO. The MuCoS event system idea is to
provide a low level interface that automatically does most of the
routing/mapping job, and allows connecting ports with minimum host
intervention. Event handling is done with inline code, with the
exception of allocating memory buffers - suballocation of individual
events is done with inline code.
The idea is to make the event system efficient enough to be a viable
alternative to the kind of ports used by LADSPA, while still keeping
it a real event system, that allows some serious work to be done on
the host side without translating everything into some other system.
That is, the event system is both an API and a protocol.
> (4) Common Event Types
>
> (4.1) In this text I use 'event' to refer to time-stamped data. I do not
> assume a concept of duration.
>
> (4.2) For different applications to share event data, they need to be able
> to interpret that event data uniformly. Perhaps before looking at LADSPA2
> or event buses we should consider a common registry for event types. Event
> type IDs could be allocated on request from a central (automated?) body, as
> we discussed for LADSPA plugin IDs.
I'm not sure this makes much sense for the MuCoS event system.
Indeed, it will have a few different kinds of events, but the mapping
to actual functions is done through channels (corresponding to LADSPA
ports) and subchannels (different functions of ports). The event
types are different ways of passing data, rather than "commands".
What needs to be standardized is how different data types are
transferred, and how data map to subchannels, ie
Q: How do I pass data to a channel that accepts "Float 3D
Coordinates"?
A: There are 3 float values, accessible through subchannels
1, 2 and 3 (x,y and z respectively). You may write these
one by one using single "Float Value" events, or two (1+2 or
2+3) at once, or all at once using "Array of Float Values"
events.
<implementation>
The simplest plugins would just set up an array of variables in the
closure (instance struct), and then index these using the channel and
subchannel numbers of received events. In the example above, the
plugin should decode "Float Value" and "Array of Float Values"
events.
</implementation>
NOTE:
I think it might be a good idea to separate the data types from the
event types, as that allows plugins (event filters, for example) to
handle events without understanding the data they carry. For such
plugins, it should be possible to decode just the event type field,
and ignore the data type field.
> (4.3) Events could have a simple single-block format starting out with a
> type_id and size - and could then be implementation specific. Alternatively
> they could use more complex data structures and provide plugins containing
> read/write methods to handle persistence/neutral transmission. The latter
> is how MN handles events but is probably somewhat over-the-top for a simple
> plugin API.
>
> (4.4) Having a common event registry could allow events to be exchanged
> freely between LADSPA, MuCoS, VST plugins and event buses, let alone the
> many hosts that could support them directly or wrapped.
Indeed, events need to be standardized to make much sense. In that
way, they're very similar to the proposed multiple datatype
extensions to LADSPA.
> (4.5) VST SDK2 has a crude concept of event type. Perhaps Steinberg would
> let us hijack this.
Hmm... I'd prefer not making the event system all that different from
LADSPA control ports + timestamped transfers. After all, if you can
transfer one or more elements of any data type from one place to
another, with a sample accurate (or better) timestamp, or an "ASAP"
stamp for out-of-band events, you have pretty much covered most
things you can possibly want to transfer, right?
No need for special event types - where you send the events is what
makes a difference. That's what allows you to connect plugins not
just in the ways explicitly planned by the developers.
As a practical example of what I mean, compare the above 3D
Coordinate example to passing 3D Coordinates as some kind of special
events. In the latter case, you need a special converter plugin if you
want to use 3 Float outputs to control such an input, while with the
design in my example, you can just connect them, using the usual
"Single Float" event, addressing the x, y and z subchannels
respectively. You may also connect a 2D Coordinate output to either
(x,y) or (y,z) without any converter plugins or remapping involved.
> (5) What do we want?
Wow! That's a good question...! :-)
> [Okay okay, what do *I* want ;-)]
>
> (5.1) Cubase running on Linux with our plugins running within it. Given the
> former, the latter will follow if ANY sensible API is provided, so this is
> probably not a problem (subject to 1.4).
>
> (5.2) Access to the vast VST plugin library. This follows if the VST plugin
> API for Linux is reasonably compatible or Windows developers are encouraged
> to do ports.
>
> (5.2) A plugin interface that can be used to share code between most Linux
> hosts. In my opinion, the current VST plugin API is *not* adequate for this
> so I think LADSPA needs to stay for the moment. Just as an example, one of
> the unit generators on my to-do list for MN is a frequency tracker. This
> would take an audio stream and apply cepstrum techniques to it to find a
> fundamental frequency. The output would be a frequency control output. I
> cannot construct this using VST as is - I'd have to use an audio output. I
> can within LADSPA or MN - or most of the other unit generator based hosts
> out there.
This is somewhat weird... VST seems to be pretty high level, and
still lacks basic features! That's the impression I got from VST 1.0,
and what seems even more strange to me is that VST 2.0 doesn't fix
it. As a matter of fact, VST 2.0 looks to me like little more than a
first-shot fix of the most serious shortcomings of 1.0.
But of course, VST wasn't designed for anything like what we're
discussing here. It's not a unit generator API.
> (5.3) An audio/event bus so that applications can work together.
I want to make this a part of the high level (ie "not unit generator")
plugin API, since that makes it a lot more useful. For starters, you
get a great deal of the UI and automation interfaces virtually for
free, as you can "directly" connect plugins or clients running in
different threads. Put another way, a plugin is just a client without
a thread of it's own - it runs as a callback function of the host
thread instead.
> (5.4) To achieve 5.2 and 5.3 nicely I think we need a common event type
> pool.
Well, yes, some sort of standard. Although, I don't know if people in
general agree that my idea of how events should be encoded is the way
to go, which indicates that there is a huge potential for an effort
of this kind ending up in lots of expensive converter stages. A
system that encodes everything in a single event type code can never
fit directly together with a subchannel based system like MuCoS...
OTOH, if the event codes are arranged in the event type/data
type/subchannel form (within say, 32 bits), those who wish to ignore
the fact that the fields can be used individually may do so - the
same codes can still be used.
> (6) What do Steinberg want? [Guesswork]
>
> (6.1) Market share.
This is what worries me the most, until *we* ("users and developers")
are in charge of the API development. Somehow, I'm having a hard time
figuring out how Steinberg could ever be to VST what Linus is to
Linux. Actually, they *can't*, unless they let go of the legal rights
to control VST.
> (6.2) They hopefully won't want to write their own audio device drivers. We
> probably should point them towards ALSA rather than OSS?
I'd guess so, yes. (OSS might not even be worth considering at the
time they release an application.)
> (6.3) They probably will want to be able to talk to other Linux
> applications, particularly software synthesisers. For this they will need
> an audio/event buses - which they probably don't want to write themselves
> although if push comes to shove they could simply use a couple of
> audio/MIDI devices and let other people connect to these - and porting
> ReWire would probably be pretty trivial but proprietary. Does anyone know
> what they did (if anything) for BeOS/IRIX?
This is too important to take the risk of getting in trouble with a
proprietary API. To serious audio developers and users, that would
be like replacing the IPC functionality with some proprietary
solution... And this is not to mention the Qt/GTK+ style flamewars
that will surely follow.
> (6.4) Innovation. (Well, I'm hoping they do - they certainly have a history
> in this direction!) Hopefully they'll be interested in sensible extensions
> to Cubase that increase its usefulness. I don't know what graphics
> libraries they used for BeOS and IRIX ports, but I cannot help but suspect
> they will have to do quite a bit of work to do a Linux port. Within this
> budget I'd hope they have some flexibility to look at interface changes,
> particularly given help they don't have to pay for! Also, there are an
> awful lot of people doing "weird stuff" (i.e. innovation) on Linux and I'm
> sure they would like these people on their side both technically and
> politically.
Yes, but there's another side of this. Supporting too powerful APIs
(in particular in the application integration area) may leave
Steinberg with less control of what users can do through simply
connecting things vs. what Steinberg can sell in order to solve
problems that users cannot solve themselves due to technical
restrictions.
It might also bring issues like "But, this application works great
with all the latest plugins! Why should I upgrade?", thus forcing
Steinberg to try to compete in whatever area the market forces them
to - which is perhaps not where they were planning to go.
Then again, I might be worrying a bit too much about not allowing more
companies to abuse power the way Microsoft has been doing.
> (7) Some ideas for Things we can Do, ideally all in conjunction with
> Steinberg
Yes, I'd really like to see those guys around here! :-) But please
keep the marketing people away from the license issues...
> (7.1) Spec out an audio/event bus. Perhaps the ALSA people would like to
> manage this although the bus probably shouldn't be part of ALSA itself.
As far as I can see now, this isn't all that different from an event
system as used for local plugin->plugin transfers. The biggest
difference is that addressing and data copying issues arise as soon
as events are sent from one process to another, or any further than
that, like from one machine to another. For example, events referring
to data buffers through pointer have to be handled copying the data
(unless shared memory is used), and reconstructing the pointer on the
other side. There should probably be ways to handle static data (like
preloaded wavetable data) in a transparent way, to avoid copying all
data all the time.
Still quite a few things to consider, that is, but I'm not sure it
makes sense to view this as a separate issue from the event systems of
the plugin APIs. There should be at least one plugin API that
interfaces more or less directly to the bus, as this results in lots
of flexibility and good performance without the price.
> (7.2) Spec out a common event type approach. I'm happy to do this - I think
> a simple approach could be fairly trivial.
After seeing the benefit of shifting from one event encoding scheme
to another, I'm not so sure this is all that trivial... Then again,
my idea above might be close to eliminating this problem.
Any other, fundamentally different encoding scheme out there, that
couldn't work with my multi-field encoding?
Just another thought on that: If the encoding scheme results in many
plugins and clients needing to use multistage switch()es to decode
the events, we have a performance problem. (switch() generates jump
tables as long as that doesn't result in huge tables with most
entries pointing at the default case.)
> (7.3) Spec out a proper unit generator plugin API that covers both unit
> generator requirements (i.e. proper control ports) and the event support
> required by synthesisers etc. This can be done by extension/rewrite of the
> VST API. It could also be done by persuading Steinberg to support LADSPA
> plugins and adding very simple event support (or using a LADSPA-alternative
> e.g. MuCoS if richer functionality is required). Otherwise MN will almost
> certainly support both LADSPA1 and VST2 plugins - this is not too difficult
> although the VST events will be a bit of a pain. I'm sure other hosts could
> handle this too - but it would be much nicer to bring things more together.
Yes, all this actually has to work, and do more than making the CPU
generating more heat. With too many wrappers and conversions there
will be no power left for actual processing...
//David
.- M u C o S --------------------------------. .- David Olofson ------.
| A Free/Open Multimedia | | Audio Hacker |
| Plugin and Integration Standard | | Linux Advocate |
`------------> http://www.linuxdj.com/mucos -' | Open Source Advocate |
.- A u d i a l i t y ------------------------. | Singer |
| Rock Solid Low Latency Signal Processing | | Songwriter |
`---> http://www.angelfire.com/or/audiality -' `-> david_AT_linuxdj.com -'
This archive was generated by hypermail 2b28 : Wed Apr 05 2000 - 06:29:32 EEST