Subject: [linux-audio-dev] Extending LADSPA to support Non-Causal Plugins
From: Richard W.E. Furse (richard_AT_muse.demon.co.uk)
Date: Thu Oct 26 2000 - 22:27:39 EEST
One way to look at the problem is to consider the causality issue
separately for control and audio data:
NON-CAUSALITY AND AUDIO
The bottom line is that in general stream-based audio processor can only
perform operations of form y[t]=f(x[g(t)]) where g(t) <= t. This means that
algorithms such as reverse (y[t]=x[R-t], 0<=t<=R) and upwards sampler-style
pitch shift (y[t]=x[t*pow(2,semitones/12)]) are not possible.
Fixes to this require a non-streaming paradigm. It has been suggested that
LADSPA could be left fairly changed so that all audio is passed in as a
single block. This works, but presents a number of problems, such as:
1. It becomes impossible to change parameter data through time.
2. It requires all audio to be available through memory. With mmap() around
this isn't a disaster, but requires the file format to be float-based.
3. This doesn't address the idea that the output audio chunk may not be the
same length as the input.
4. The plugin is not usable in a streaming context. This is a fundamental
problem with non-causal audio.
Point 4 means that the plugin will not be usable in current (stream-based)
hosts so there is no fundamental need to maintain backward compatibility
for non-causal plugins. I have some ideas of ways to make additions to the
API allowing maximal code reuse without steepening the learning curve too
much or breaking existing hosts/plugins. See below.
NON-CAUSALITY AND CONTROL DATA
For many plugins, causality issues with envelope shapes can be got around
by mediation by the host, which DOES know exactly what will happen in the
future. As a good example, there are two ways to look at a 'fade in'
effect:
1. This could be modelled using a non-causal model and implementing using
an analogous approach to the one described above for audio. I don't like
this very much.
2. The fade in effect can be thought of as an enveloped application of a
gain plugin. This requires more sophistication in the host as the host has
to be able to encapsulate 'patches' assembled with envelope generators and
streaming LADSPA plugins.
I think case 2 is roughly Conrad's model and was how I was thinking about
this initially. It seems like overkill for fade in, but this kind of
functionality allows clever stuff such as turning a formant filter into a
speech synth that actually says things. This is very nice functionality and
I recommend that hosts support this even if going with a model capable of
handling the more difficult non-causal case.
TOWARDS AN API FOR NON-CAUSAL PLUGINS
Accepting that conventional LADSPA is essentially causal and that
non-causal filters are essentially incompatible, I'm thinking the best
option might be to extend the API in the following broad way:
1. Leave the plugin structures returned by ladspa_descriptor() the same for
backwards compatibility.
2. Add ladspa_noncausal_descriptor() to grab non-causal plugin descriptors
from a library.
3. Add run_noncausal() and run_noncausal_adding() function pointers to the
structure. These would be used INSTEAD of the existing run() and
run_adding() functions for non-causal plugins and would have a slightly
different prototype (returning a long). It might actually be possible to
use the existing slots in the LADSPA_Descriptor structure - the C experts
out there might have a view on this.
4. Add a connect_transport() function call which must be called before
activation (at least before the first activate() call). More below.
5. Output ports would be described as before and would work as usual, with
a slight twist.
6. Input ports would be described as before with a new flag
LADSPA_PORT_NONCAUSAL and would work as before when the flag was not
present. LADSPA_PORT_NONCAUSAL introduces new behaviour.
7. New plugin-level flag LADSPA_PROPERTY_REQUIRES_BOUNDED_INPUT_LENGTH.
8. Things otherwise stay as before, the plugin is used much as before.
MORE DETAIL
It will become very common (although not necessary) to see the flag
LADSPA_IS_INPLACE_BROKEN.
The smallish change to the run() methods and output ports is follows:
1. If the run() method returns a number less than the number of samples
requested then assume that only that many samples have been output to the
audio output buffers and read this as "end of output."
2. On the end block, the output block may not be entirely filled. The
unused area will not be written to by the plugin. (The host may wish to
zero it etc, may have already done so for _adding() etc.) If the end block
contains 0 samples then control outputs may not be written to.
The clever bit happens through the new connect_transport() function. This
would work something like this:
[...]
/**********************************************************************
*******/
/* Non-Causal Plugin Input Transport Control:
This structure only applies to non-causal plugins.
This structure provides allows the plugin to control the input data
provided to it at runtime. It is provided through a call to
connect_transport() (see below). */
typedef struct _LADSPA_InputTransportControl {
/* The number of samples of input data available. This may be zero
indicating that there is no limit to the input data available -
although
this will not work with all plugins (e.g. reverse). Setting this to
zero
is prohibited for plugins declaring property
LADSPA_PROPERTY_REQUIRES_BOUNDED_INPUT_LENGTH. */
unsigned long InputLength;
/* A function that allows the plugin to move the reference point on all
ports marked as LADSPA_PORT_NONCAUSAL and also change the amount of
audio present in the buffers. For instance, a reverse plugin
working on its first block of SampleCount samples would set the
reference point to (InputLength - SampleCount) and ask for
SampleCount
samples.
This function only affects input ports labelled
LADSPA_PORT_NONCAUSAL.
The function returns the number of samples of input that
could be provided - this must be positive but may be less than
SampleCount,
for instance because the host wishes to change control input
settings. The
host must not read beyond this point.
The host may (or may not) call the connect_port() method on the
plugin during a call to move_noncausal_inputs().
This function must be called after activation or reactivation
and before input data is referenced for the first time inside run()
calls. After this the plugin is free to call this any number of times
per run() call until deactivation. If it is not called at all the
host
should leave data as is. */
unsigned long (*move_noncausal_inputs)(unsigned long Offset,
unsigned long SampleCount);
} LADSPA_InputTransportControl;
/**********************************************************************
*******/
[...]
typedef struct _LADSPA_Descriptor {
[...]
/* Connect a LADSPA_InputTransportControl structure to the plugin.
This only applies to non-causal plugins. (See the
LADSPA_InputTransportControl structure above.)
For a non-causal plugin, this call must be made before an
initial call to activate(). If the plugin is deactivated then
the host may (or may not) choose to call before activate() is
called again.
The transport structure may not be changed by host or plugin
while connected. */
void (*connect_transport)(const LADSPA_InputTransportControl *
Transport);
} LADSPA_Descriptor;
/**********************************************************************
*******/
[...]
This gives the plugin the ability to choose to have its inputs work as
usual, or alternatively to access any part of the input stream. Output is
much as usual, any existing code that publishes/presents hints, controls
and descriptors continues to work for all sockets except non-causal inputs.
And these are necessarily so changed that one would expect them to break.
The host has to support the transport mechanism.
I think this is nice and even allows non-causal plugins to stream output in
real-time (assuming the host can provide the non-causal inputs fast
enough).
What do other people think?
--Richard
This archive was generated by hypermail 2b28 : Thu Oct 26 2000 - 23:07:07 EEST