Subject: [linux-audio-dev] An alternative API proposal for LAAGA
From: Jim Peters (jim_AT_aguazul.demon.co.uk)
Date: Sat May 12 2001 - 00:47:15 EEST
It occurs to me that everything we've discussed so far has been based
on Paul's work with AES. Because of this the discussions have all
been in reference to this work, for or against it, or based on making
modifications to it. However, no-one's really come up with a complete
alternative to actually compare with.
For this reason I'm trying to do that here. The point of this is not
to create competition or anything like that. Rather it is to provide
some additional material to build on. Probably we'll end up with some
features included here, some features from AES, bits from here and
there. Really I wanted to get all my design ideas down so that this
can be used as a resource to borrow from.
Also, I'm putting together here ideas from many directions - things
people have mentioned on the list, ideas we've developed in discussion
- so most of this is already in the list archives. Still I think it
is worth putting this all together into a single proposal, whilst it
is all still fresh.
Paul has posted a big summary since I started writing this which I
haven't studied yet, so there may be some duplication, but I wanted to
get this finished first whilst it is current in my head.
Note that here I'm just covering the audio part of the server and
plugin communication. Transport controls, time-code or whatever will
have to come later. Also, I'm not discussing how external processes
communicate with the server either.
From here on down `the server' refers to the one I've got in my head,
not necessarily the one everyone else is thinking of.
Proposed server features
------------------------
It's worth noting the following features of the server:
- There will be one real-time thread in which all critical audio
processing will take place. No blocking operations or system calls
are permitted here.
- There will be one server management thread which will take care of
management issues, such as setting up plugins and ports, reordering
the run list, making and breaking connections and so on. It also
takes care of running submitted jobs whenever required (see later
for details on this).
- There may be other threads in the server which belong to particular
plugins. These may be used for whatever is necessary - disk i/o,
blocking on sockets, and anything else required.
Basically, I see the server management thread having exclusive rights
to dealing with operations on plugins, ports and connections.
Operations on these will be processed serially by this thread. This
avoids problems of locking and races and so on. This does however
mean that if a plugin needs to make changes to ports or connections,
it has to run its code in the server management thread. The plugin
entry point `plugin_job_handler' and the server entry point
`submit_job' are designed to permit this (see below). In addition,
convenience calls to schedule common operations are included:
`add_port_quick' and `del_port_quick'.
Other features:
- The real-time thread operates in `cycles'. Each cycle consists of
reading a chunk of input data from the sound hardware, running all
the plugin_process() routines, and finally outputting to the sound
hardware. This cycle-period defines the latency of the server.
- The real-time thread's primary purpose is to keep up with the
hardware, so if a plugin's plugin_process() routine hasn't got data
available to output, or if it can't find anywhere to store its
input, then it must take some appropriate action and return, not
block or wait. What is appropriate depends on the plugin's task -
for some it may be okay to substitute silence, for others it would
be better to flag an error and disable themselves. Whatever is
decided, I ask plugin-writers to make sure that small errors can't
go unnoticed. So if you have to substitute silence, substitute
100ms (not 1ms), and flag the error somewhere where the user will
notice it.
The plugin entry points
-----------------------
The plugin provides the following entry points:
void *handle= plugin_new_instance(char *args);
plugin_process(handle, int nframes);
plugin_job_handler(handle, int arg1, void *arg2);
plugin_delete_instance(handle);
All of these calls except `plugin_process' are called from the server
management thread, and they may perform all kinds of system calls as
required, setting up threads, shared memory and so on. The only
restriction is that if they take too long, they'll hold up other
server management functions - these calls are intended only for setup
and configuration types of operations. If the plugin needs a lot of
time for setting up (indexing some directories, say!), it should start
its own thread to handle this rather than holding up the server
management thread. This won't affect on-going operation of the
real-time thread in either case. `plugin_process' on the other hand
is only called from the real-time thread, and it must take care not to
make any system calls or take other actions that could block.
`plugin_new_instance' is called to do initial setup for the plugin.
The void* returned is intended for the plugin to allocate itself a
structure to contain its state, although use of this is optional.
`args' is a string that was passed to the server when it was requested
to load the plugin (by the GUI or whatever). This may be used to pass
info to the plugin - for example info may be passed to help the plugin
get its communication link setup with its GUI. This routine should
call the `set_name' function below at some point to name the plugin.
`plugin_delete_instance' is expected to release whatever allocations
of resources the plugin still holds, whether these be from
`plugin_new_instance' or from later allocations. This includes
terminating any threads that it has running. It is important that all
resources are freed as this server will be long-running and resource
leaks will accumulate otherwise.
`plugin_process' is called from the real-time thread, and may not
perform any type of system call or blocking activity. Communication
with the outside world should be via shared memory or internal memory
buffers shared with other threads in the server (for example, ones
that the plugin started during its `plugin_new_instance' call). It is
expected to process `nframes' of audio between its input and output
buffers (see below), which will have been setup to point to the
correct buffer locations by the server. It should not modify input
buffers, because these may be reused for other plugins. It should add
its output to the existing contents of the output buffers,
run_adding()-style.
`plugin_job_handler' is called from the server management thread to
handle a job triggered when the plugin calls `submit_job' from one of
the other threads. The arguments are those passed to `submit_job'
below. This mechanism is used to allow a plugin to execute some
management-related code whenever necessary.
Server entry points for use in the management thread
----------------------------------------------------
The following calls are available to be called from within the
management thread - i.e. within the plugin_new_instance(),
plugin_job_handler() and plugin_delete_instance() routines. These
should not be called from the plugin_process() routine or from other
plugin threads.
int okay= set_name(char *name);
This is used to set the name of the plugin. It should only be called
in the plugin_new_instance() routine, and never again after that. It
may return failure if the name is already in use. This name will be
used for debugging, and as a unique identifier for the plugin for
making connections.
add_port(handle, float **floatp_varp, int direction, char *name);
del_port(handle, float **floatp_varp);
`add_port' adds a new port to the plugin instance, either input or
output as indicated by the `direction' argument. The plugin must
provide the address of a float* variable into which the buffer pointer
will be placed by the server. This variable can be anywhere in memory
owned by the plugin - for example, in a global, or in part of its
state structure, or in some other allocated structure or list. The
server may change this variable's value before every call to
plugin_process() if it wishes. Using a float* in this way should
allow the maximum convenience to the plugin, to fit in with whatever
internal system it is using to manage its ports.
`del_port' deletes an existing port based on this float* address.
int okay= connect_port(char *src_plugin, char *src_port,
char *dst_plugin, char *dst_port);
This attempts to connect the output port on one plugin (src_*) to the
input port on another (dst_*). Maybe these four arguments could be
replaced by two if a standard plugin-port separator ('-' ?) can be
agreed upon. It fails only if one or the other of the ports does not
exist.
Feedback loops are handled automatically by inserting a delay equal to
the cycle-period into the connection that is part of the longest path.
The audio hardware will appear as two separate plugins with different
names - one which has just outputs, and the other which has just
inputs. See my little Perl-script from a previous posting for a rough
idea of how I see this.
int okay= disconn_port(char *src_plugin, char *src_port,
char *dst_plugin, char *dst_port);
Similarly, this disconnects two ports, returning false only if the
ports don't exist or if the connection has already gone.
There will also have to be a whole bunch of routines for querying the
list of loaded plugins, their active ports and the connections between
them. I've not listed them here because they are less interesting
than the rest of the design right at this moment.
Real-time server entry points
-----------------------------
These calls are designed to be made from the real-time thread -
i.e. from the plugin_process() routine. They are designed to take
very little time, handing off major tasks to the server management
thread. These calls can also be used from plugin-specific threads.
submit_job(handle, int arg1, void *arg2);
This allows the real-time thread to trigger some action in the server
management thread. This is indended for management operations, and
also for any other bits and bobs of processing that the plugin
requires, but that can't be performed in the real-time thread due to
time constraints. This call may also be used from other plugin
threads to initiate management tasks.
A short time after this call, the server management thread will call
`plugin_job_handler' with the provided arguments. I chose an int and
a void* as arguments on the basis that these would be able to handle
nearly everything that might be passed (i.e. an enum, or a string, or
a structure or whatever), although I'm open to suggestions here.
Regarding exactly when the `plugin_job_handler' will be called, it is
likely to be very soon after after the `submit_job' call, because the
server management thread will be woken if it is sleeping. It cannot
be guaranteed that the call-back will come right away, though, because
the server management thread is not real-time, and there may be a
couple of other jobs in the queue to clear first in any case. I don't
anticipate this being a problem.
add_port_quick(handle, float **floatp_varp, int direction, char *name);
del_port_quick(handle, float **floatp_varp);
These two calls are exactly the same as the ones without _quick above,
except that these can be called from the real-time thread or from
other plugin threads. The action is not performed immediately.
Rather a dummy buffer is put into the float* variable (either a
standard all-silence buffer for input or a standard dev-null scratch
buffer for output), and the change is submitted to the server
management thread for execution soon after. The plugin may start
reading or writing to the port right away, but the port won't actually
become `visible' until the management thread gets around to processing
the request.
Other real-time resources:
-------------------------
These are global variables available to read from any thread.
int time_now;
This is a frame-counter that is updated at the start of each cycle.
This can be used by the plugins for synchronization, transport
operations and so on, once these are defined. This counter rolls over
every 12.4 hours at 96kHz. We could extend it to a 64-bit int if that
is seen as a restriction.
int frame_rate;
This is the sample rate at which the server is operating, in Hz. This
is not going to change during the life of any plugin instance. If the
sample-rate changes, then all the plugins will be shut down -- the
whole server will restart.
Notes on implementation of buffer operations:
--------------------------------------------
There are several approaches I'm aware of to reducing copying as data
is passed from plugin to plugin, and eventually to the hardware. The
initial choice is between a run()-style method (that is, having the
plugin write directly to the buffer: buf[i++]= val) and the
run_adding()-style method (having the plugin add its output to the
existing buffer contents: buf[i++] += val).
I don't think it is feasible to use both the run() and run_adding()
methods in LAAGA because, unlike a LADSPA plugin, these plugins are
full applications often with many ports. Switching all of these ports
to run()-style or all to run_adding()-style won't really help the
server much when it probably wants run_adding() for some and run() for
others. So better forget this idea for LAAGA, I think. We need to
choose one or the other.
Here is a simple comparison of purely run()-style versus purely
run_adding()-style:
- Purely using run() is better for systems that are dominated by
one-to-one connections (output port -> input port), or one-to-many
connections (one output port -> many input ports), because it saves
the time required to zero the buffer in the run_adding() case.
- Purely using run_adding() is better for systems that are dominated
by many-to-one connections (where many plugin outputs go to a single
plugin input), because it saves an extra stage for mixing as would
be required for the run() method in this case.
There is an additional optimization that is possible if the
run()-style method is used. This allows passing data through a plugin
without copying. It works like this: A plugin is provided with output
buffers, and if it wishes it can fill these buffers as normal.
However it may alternatively copy across an input buffer pointer to
its output buffer variable instead, effectively passing a whole buffer
from input to output in one step. The server will pick up whatever is
pointed to by this variable as the output of the plugin.
This method allows very fast transfer of data from input to output of
a plugin, without copying, but it does mean that the server can't
predict which buffers will become free (as any input buffer might be
passed on as an output buffer), reducing its ability to optimise for
caching. Whether this method is useful or not depends on whether
plain copy-through transfers will be a significant proportion of the
work done by plugins in the system.
This gives three alternatives, as far as I can see right now. The
proposal above is based on option (1):
(1) Use purely run_adding()-style method: buf[i++] += val
(2) Use purely run()-style method: buf[i++]= val
(3) Use run()-style method, but also permit copying buffer pointers
between input and output.
Here are some examples to compare (1) and (2):
- One-to-one using run()-style method and buffer X
execute plugin 1 ()->(X)
execute plugin 2 (X)->()
- One-to-one using run_adding()-style method and buffer X
zero buffer X
execute plugin 1 ()->(X)
execute plugin 2 (X)->()
- Three-to-one using run()-style method and buffers X Y
execute plugin 1 ()->(X)
execute plugin 2 ()->(Y)
mix buffers (X,Y)->(X)
execute plugin 3 ()->(Y)
mix buffers (X,Y)->(X)
execute plugin 4 (X)->()
- Three-to-one using run_adding()-style method and buffer X
zero buffer X
execute plugin 1 ()->(X)
execute plugin 2 ()->(X)
execute plugin 3 ()->(X)
execute plugin 4 (X)->()
Optimisation of the run-list and intermediate buffers by the server
-------------------------------------------------------------------
The server will take the list of plugins and the list of connections
and devise an order of execution that makes sure that all the input to
each plugin is prepared before that plugin is run.
It can also optimise the use of buffers so that unnecessary copying is
reduced as far as possible. For example, if one plugin's output port
is connected with others to another plugin's input port, then using
method (1) it can mix straight into that buffer. However, if the
plugin output is also connected elsewhere, it must write to a separate
buffer, and the server will take care of mixing that data into the
destination buffers if necessary.
The plugin-writer doesn't need to know anything about this, as the
server can optimise all of this correctly.
...
Okay, that's it. Brain-dump complete. I expect that there will be
objections to some of the things I suggest. I'm not attached to
anything I've proposed here - I'm only attached to getting something
that will do the job, and that we can agree on. To me these are all
ingredients we can rearrange until we get exactly the mix that we need
for our purposes.
Jim
--
Jim Peters / __ | \ Aguazul
/ /| /| )| /| / )|| \
jim_AT_aguazul. \ (_|(_|(_|(_| )(_|I / www.aguazul.
demon.co.uk \ ._) _/ / demon.co.uk
This archive was generated by hypermail 2b28 : Sat May 12 2001 - 01:00:34 EEST