Subject: RE: [linux-audio-dev] ladspa GUI round 2
From: Richard W.E. Furse (richard_AT_muse.demon.co.uk)
Date: Tue Apr 03 2001 - 01:03:18 EEST
Ooo - bad week to be behind on my email! Some thoughts:
REQUIRED INTERFACES
I agree communication between GUI plugin and audio plugin should use the
following path:
[GUI Plugin] <-a-> [Host] <-b-> [Audio Plugin]
In this email, <-a-> indicates the link between GUI plugin and host, <-b->
indicates the link between host and audio plugin, <-c-> indicates any link
between host and display API.
This is sensible for automation and remote use. The existing LADSPA API
supplies an implementation of the <-b-> link that is simple but isn't too
broken (IMHO). I don't think the audio plugin should be told whether or not
it is attached to a GUI plugin - indeed I think if the GUI plugin needs to
know whether or not it is attached to an audio plugin then something is
probably wrong. It isn't necessary to provide remote functionality in
either <-a-> or <-b-> as a host can provide the remote bridge itself.
There is a second interface required, between the GUI Plugin and Display.
This could be abstracted using the VST and XML approach or allow direct
calls to the UI layer.
[GUI Plugin] <-a-> [Host] <--> [Display]
or [GUI Plugin] <-a-> [Host]
[GUI Plugin] <-c-> [Display]
I must admit to a bias towards the former. More later.
CONTROL VIA OSC, MIDI ETC
I'm a little nervous about insisting on use of OSC. If it is enough of a
standard this is fine by me, but it sounds a little heavy handed (string
passing is not efficient and folk might want to transmit over a WAN). And
anyway, if we used a simpler interface for <-a->, e.g. C or sockets or IPC,
it would be trivial to write a wrapper to perform
[GUI Plugin] <-a-> [GUI host] <-OSC-> [Audio host] <-b-> [Plugin]
I've not looked at OSC so I don't know if it handles plugin start up,
shutdown, audio streaming etc. If it does then we could do the above now
for a GUI host using Paul's XML spec :-)
This kind of approach is also compatible with control transmission using
MIDI or suchlike, although obviously standards would have to be specified
for this to work in general. (Hmm we probably ought to spec up a MIDI Sysex
protocol for controlling a remote LADSPA FX processor and synth.)
RELATIONSHIP BETWEEN AUDIO AND GUI PLUGINS
It seems to me that the most natural mapping between these is to use the
audio plugin ID as the link key. Audio plugins have unique IDs already, and
plugins can indicate support for a particular plugin explicitly in this
way. There is nothing to stop support for multiple GUIs - as suggested,
this might be useful when beginner and expert modes are available A user
might even like to write his/her own 'skins'. It is also useful if
different hosts have differing GUI capabilities - a host may not be able to
support XML GUIs but be quite happy with Qt or GTK etc. The latter comment
isn't supposed to pre-empt discussion of...
HOW THE GUI IS HOSTED
This was the sticking point as I remember it previously. I gather some of
these issues have been addressed, although I'm not clear of the conclusions
from the list. I'm not a Linux GUI expert (I still do most of my coding and
music in emacs on a console screen). I ask for information rather than to
be difficult - questions:
1. How does the host know what GUI plugins are available?
2. How does the host wire the plugins, start/stop them, send/receive
control values etc? (Paul's suggestion and OSC and a number of other
approaches could work as a basis here.)
3. When does everything happen? How does operation differ when running
offline to when running live?
4. Does the plugin exist within the host's GUI or externally? ... and how?
Some approaches that spring to mind, all broken in some way, but perhaps
someone can fix one or more:
A [auto] GUI plugins can be generated automatically by the host from
existing LADSPA plugins. Existing graphical hosts do this to the best of
their ability, but there are limits to how far this can be taken as the
host has no fundamental understanding of the meaning of the data on control
ports.
B [XML]. An XML approach already exists and can work well for audio plugins
that do not require any special displays, breakpoint graphs, multiband
meters etc.
C [wrapper]. Plugins exist in C library files using dynamic loading and an
API very similar to the LADSPA audio one (i.e. includes calls like
connect_port()). This could be blindingly fast for self-contained
applications as the audio and GUI plugins could use the same memory address
to communicate control values. Remote applications could use
OSC/sockets/MIDI/something proprietary or whatever as a remote
communication protocol (see Control Via OSC above). The link between GUI
plugin and host could be via a standardised GUI wrapper API (VST does this
sort of thing). Different hosts could have their own look-and-feel and even
a rackmount Linux box could make a stab at an implementation. The problems
with this are
i. Thread locking issues need to be resolved (probably not too hard and
some hosts might prefer to be single threaded, e.g. for SoundForge type
applications).
ii. High implementation cost for the host.
iii. Learning curve for the GUI designer (although maybe this is
inevitable).
iv. The API might become a maintenance problem. Adding cool new widgets to
the API would break old hosts - but the next generation of programmers is
bound to want more. (Perhaps we need a GUI widget API. Eek.)
D [native]. For the sake of having it listed, the above, but using GTK/Qt
or suchlike rather than a VST-type wrapper interface. This is very broken
IMHO as non GTK/Qt (or other) hosts cannot use the GUI plugin. (Come to
think of it, Java is an interesting alternative as it could enable remote
control on a web page...) However, see F and G below...
E [executable]. Plugins exist in a common directory as binary files that
can be run to extract parameter/configuration information and then executed
by hosts locally or remotely. Communication is then maintained using OSC,
IPC, sockets or pipes to stdin/stdout. This could be quite fast and makes
the multitasking aspect explicit. Problems:
i. The overhead for host and plugin design might be quite significant
(although a library of standard plugin code could be provided).
ii. Relatively high programming complexity.
iii. Different plugins will have very different look-and-feel and using a
number will make the plethora of Windows confusing rather quickly. (Which
one is wired to which? Which compressor is associated with track 19? etc.
etc.)
iv. Using lots of plugins will require lots of processes. Linux
multitasking is good, but I'd prefer to keep the GUI down to one or two
threads, not one per plugin. Also, how many GUI *instances* can be provided
by one process? If many this is harder for a naive programmer to work with.
If just one then even more processes will be required. Just think what a
mess your 'top' will become :-)
v. Potentially high start-up cost (relatively).
vi. Makes me nervous for other reasons I've not put my finger on yet.
F [meta]. Some kind of 'meta plugin' approach where a dynamically loaded
library (or config file) provides information about the plugins available
using a number of different standards (e.g. a really OTT plugin library
might contain the LADSPA audio plugin itself, an XML description of a GUI,
a native GTK GUI, a native Qt GUI, a VST-type GUI and the name of an
executable or perhaps just a call that fork()s internally). The host could
then pick a GUI approach that suited it and get to work. If no GUI did suit
it, a default GUI could be generated using the old-style audio plugin's
description of itself. Problems:
i. General compatibility requires a lot of work from the plugin designer
and/or host.
ii. Potentially difficult to maintain standards etc.
iii. Potentially high complexity.
G [variety]. Accept diversity and work with it. Allow more than one of the
approaches above and let the host and plugin designers sort themselves out.
Problems much as for F, although the approach would be simpler and more
flexible, at least initially. Would need careful co-ordination or could get
completely out of control.
I suppose my leaning is for some variant on C [wrapper] with a wink in the
direction of G [variety], keeping B [XML] very much alive.
SOME TEST CASES
A working API should ideally handle all the following (somewhat random)
cases. Standardised wrappers for the existing LADSPA layer are required for
a couple of them (e.g. a MIDI Sysex convention), but nothing controversial.
Some hosts (imaginary, any similarities coincidental etc etc):
1. Rackmount 386 FX processor with custom low-latency kernel, small
graphical LCD display and no X-Window.
2. The above, but without the LCD display. Has MIDI though...
3. Single-threaded sound editor written in Lisp. Regions will be selected,
plugin GUI will pop up with the parameters used last time, plugin GUI will
be edited, the GUI will vanish and the audio will be processed according to
the parameters set.
4. A 'remote' host exists on a DSP card built into the PC. Plugin IDs are
used to identify algorithms in the usual way although the DSP libraries
would go into a different directory (a PC might even have two
implementations of a plugin, one compiled for DSP, one for CPU).
5. Flashy multithreaded audio sequencer with automation. Most parameters
are fixed on the GUI, then others are overridden using 'continuous'
Cubase-style graphs of control values that can be viewed and 'painted'. The
GUI will update live as the sequence is played back and live tweaks to the
GUI can be recorded.
6. The host system has a core that runs on a remote Sun box while its GUI
component runs and displays on a local machine over a busy network (while
streaming audio & video for a gig).
Some plugins:
5. Audio plugin is a 512 channel spectrum analyser, GUI plugin is to
display these on meters.
6. Audio plugin is a synth with four different FM configurations and eight
envelope generators, all of which require graphical editors with five
breakpoints.
7. Audio plugin is a complex general purpose filter with two GUI plugins.
The first GUI plugin has just a cutoff frequency and resonance and sets the
filter coefficients to operate as a Moog-style resonant low-pass filter.
The second GUI plugin presents vowel diagrams to the user and sets the
filter coefficients to generate the appropriate formants.
I'm sure some more pathological cases can be thought of...
THE END
Apologies for rant.
--Richard
This archive was generated by hypermail 2b28 : Sat Apr 07 2001 - 16:00:30 EEST