Subject: [linux-audio-dev] Draft: Audiality article for MusicStation
From: David Olofson (audiality_AT_swipnet.se)
Date: ma loka 04 1999 - 16:25:53 EDT
Here's a first draft of my article. As this is more about the plug-in API than
about the project I actually started up a few months ago, I'd really like some
comments from the ones involved in the API discussion.
I think this article is a bit too technical to get the "political" message
"Design Open Standards and *use* them" across, and I think that actually
belongs in a separate article. Is there actually a need for such a document, or
is the problem more of the kind "How do we get people to actually participate
in the design discussions?" It seems that there's no problem getting people to
realize that standard interfaces makes code more reusable (unless the
interfaces are crap, that is), but it's harder to get the work done. Hacking is
more fun, right?
Of course all comments are welcome! :-) And keep in mind that this document (the
first part, at least) is meant to make sense to end users - not just audio
hackers.
//David
---8<-------------------------------------------------------------------
Audiality
-------------------------------------
What's Audiality all about?
===========================
The name "Audiality" was constructed from the words "audio" and "reality",
and turned out to be very handy, in that it's unique and not used anywhere
on the Net. As might be guessed from the origin of the name, it's about
audio.
While starting out as yet another Windows application, doomed to suffer from
Windows' poor real time performance, Audiality has since been dropped and
restarted as a similar project for Linux/RTLinux. Apart from the new
environment, the restart project also has a slightly different goal: Rather
than Audiality being a stand-alone application, it's now being designed from
the ground up as a generic audio/MIDI/multimedia engine, or rather server,
optimized for reliable, high accuracy, low latency real time processing.
Currently, it seems to be evolving into an open design/standardization
effort, where a plug-in API and a client/server API currently have higher
priority than the actual implementation. The implementation itself may
actually end up as a part of some other project.
What does it mean for the user?
===============================
Basically, the success of Audiality would mean that most future development
efforts in the Linux multimedia area would result in plug-ins that can be
used by most applications, and/or applications that can cooperate on a level
that is not possible with any currently existing systems. The client/server
model allows fine grained integration all the way down to intermixing
plug-ins managed by different applications, and even controlling individual
plug-in parameters from different applications.
Everything will automatically stay in sync, as all time critical work is
done centrally, inside the engine, or is done using the same time base.
Further, all processing that is done inside the engine can be done with
very low latencies, as all plug-ins are executed as a part of the engine
thread. (Or threads, on multiprocessor systems.) Currently, that means less
than 5 ms with Ingo Molnar's low latency patch, and less than .5 ms with
RTLinux. (The later is actually a hardware restriction of most PCI sound
cards - RTLinux itself allows latencies lower than .05 ms even on Pentium
CPUs.)
As both Ingo's patch and RTLinux provide hard real time characteristics
(that is, guaranteed maximum response times), the reliability is in the
same class as that of dedicated hardware systems. The processing is done
reliably with the above mentioned latencies, as long as the operating
system stays alive. And Linux is pretty good at staying up, as you
probably know!
What does it mean for the developer?
====================================
Most importantly, easier code reuse. Plug-ins don't even have to be
recompiled to be reused with new applications. Although the recommended
method is letting the central engine/server host the plug-ins, and use the
client/server API to control them, applications may host plug-ins locally,
which means the plug-ins can be used in environments without the server.
(Note: The APIs are under heavy discussion on the linux-audio-dev list,
and some details mentioned below may change before the final standard is
agreed upon.)
Client/server architecture
--------------------------
The most important advantage of running the plug-ins in the server is that
the application can take advantage of the performance provided by Linux
SCHED_FIFO, or even RTLinux - without the need for complicated real time
code in the application. And of course, you don't have to implement a
plug-in host yourself. Another important advantage is that multiple
applications may do this at once without the reliability problems caused
by multiple real time threads running at once.
The client/server API also allows things like streaming to/from applications
(which means you *can* have applications directly processing data if you
really don't want to do it the right way; hack a plug-in ;-), sample
accurate timing with low communication overhead (events are passed in
buffers, or through pipes, much like the audio data) and off-line
processing. It's likely that a linked library containing a full
implementation of the engine will be available, which would be very handy
when porting applications to environments lacking a working server
implementation.
As an extra bonus, the server will provide OSS and ALSA interfaces, and will
act as a sound daemon for those, hiding the real devices from applications.
This will make old/non-Audiality applications work, and even fit nicely into
the system - their outputs and inputs will show up as extra ports inside the
processing net of the engine.
Plug-in architecture
--------------------
The plug-in API is OS independent, and a basic requirement for normal
plug-ins (as opposed to "system" plug-ins, like interfaces and drivers) is
that they don't do any system calls. The plug-in API is the only interface
available, and covers all things that makes sense to do within the real
time part of a plug-in. (The user interface part of plug-ins is different
in that it *allows* any other APIs to be used, and only adds part of the
Audiality plug-in API for communication with the engine and the processing
part of the plug-in.)
The API is rather low level, and provides very few function calls. Many of
the details are handled by macros and inline code for performance reasons,
much like things are done inside the Linux kernel. The plug-in API itself
doesn't know much about audio data formats, and can handle other kinds of
data, like video and compressed formats. It can be seen as a real time IPC
system, optimized for low latency, real time streaming.
The event system
----------------
One important part of the plug-in API that also shows up in the
client/server API is the event system. It allows sending *buffers* of
timestamped events (as opposed to sending one event at a time through a
function call), and supports request/reply schemes for higher levels of
communication. Events are allocated dynamically, and can have different
sizes. Timestamps are normally in samples from the start of the current
buffer, but it's likely that higher resolution formats will be supported.
Design and performance
----------------------
Plug-ins are callbacks - not individual threads as in some other systems.
Currently, this is the only sane way of doing it, for performance and
complexity reasons. Running plug-ins as individual threads would make it
virtually impossible for the engine to keep track of who's doing what and
when, and also means far too much context switch overhead, even for
RTLinux.
Plug-in scheduling is based on sub nets with homogenous cycle times, which
means that the inter-plug-in communication and streaming can be efficiently
optimized. All plug-ins within a sub net will execute one at a time, in a
fixed order.
The language of the API is C. C++ would add more complexity and hidden
overhead than it would simplify development, as a single plug-in is a very
simple, low level object.
Relation to OSS and ALSA
------------------------
This is actually implementation dependent, and has nothing to do with the
Audiality APIs. However, a server/engine implementation is likely to run on
top of the OSS or ALSA API, and as mentioned above, there will be interfaces
that allow non-Audiality applications to communicate with the server.
Audiality and RTLinux
---------------------
When the Audiality project was started, the only option for reliable low
latency processing was using the RTLinux kernel patch. A while later, there
was a discussion on low latency audio processing and real time programming
in general on the linux-kernel list, and shortly after that, the low latency
patch made low latency processing using SCHED_FIFO scheduling possible. This
will make things easier for users and developers, as production systems can
actually run high end multimedia applications in user space.
The RTLinux implementation is still part of the plans, and will be an
alternative for applications that demand extremely low latencies. A driver
API, the Driver Programming Interface (DPI) is in development, and will make
it easy to port standard Linux drivers to RTLinux. (Source can be downloaded
from the Audiality site.)
David Olofson <audiality_AT_swipnet.se>, 1999
---8<-------------------------------------------------------------------
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:27:12 EST