Subject: Re: [linux-audio-dev] Newbie
From: David Olofson (audiality_AT_swipnet.se)
Date: ke loka 20 1999 - 19:36:52 EDT
On Wed, 20 Oct 1999, MOULET Xavier CNET/DMR/ISS wrote:
> Hi David & all,
>
> I used to want to write a complete audio linux thing and when it came down
> to the architecture of the audio core, it all boiled down to the same
> conclusions as you did .. then i checked your mailing list and your
> excellent and great project. Wow. I'm impressed.
>
> As a matter of fact, koobase is now Brahms (for the links of the web page)
>
> I just joined the list and am impressed of the level of the discussion.
> I think great things are gonna happen here.
>
> May I ask a few question ? Maybe the thing I will write have already been
> discussed, maybe it would be worth reading.I take the chance. Sorry for any
> inconvenience/dumb question.
>
> 1) Is there a draft of the API/architecture ... Is there a FAQ ?
Nope, not yet. I'm going to gather the ideas we have discussed so far
into a document of that kind, and post it for comments and additions,
before I redirect the console input from the mail client to the code
editor for a while.
> (Following to be read if answer to 1 is No/No ;^) )
Ok, I'll read on, then... :-)
> 2) I understood it would be only a CORE engine, a la (esd (which actually is
> a mixer), the aRts core, or more accurately Be Media ? (I checked the Be
> BOOK, here are excellent ideas to get inspired by ...). Just a thing ..
> would it be coded in C ? C++ ? COBOL ?
First, the point isn't the engine, but rather the *API*. The main
goal is to create a standard API, so that we can get rid of the
current 110% incompatibility problem we have now - converting
plugins from umpteen API's when they could all use the same one.
It'll be binary compatibility.
The API will be C. It's too low level, and not complex enough to
justify C++. The performance requirements are rather extreme, as we
don't want people to avoid the API for being too inefficient, so the
abstraction that C++ so nicely supports cannot be used much anyway.
(Virtual member funcs for example - we won't do function calls *at
all* from within plugins in the average case, not even direct ones!)
(BTW, my Audiality project is kind of asleep, except for some
occasional hacking on the Driver Programming Interface, and of
course, the API discussion.)
> 3) Do you really intend to rewrite ALL the ALSA drivers ? isn't it a too
> hard task ?
Hmm... First of all; we don't really need to. The low latency patch
provides BeOS class performance in normal Linux user space, so engine
implementations can use whatever drivers they like.
However, it's not unlikely that Jaroslav will find some parts of our
API useful, and use them to improve his design further. That could
mean we get hardcore event communication right into the driver
subsystem (ALSA shows up as a client), which would mean sample
accurate sync with multiple sound cards, and other groovy high end
features. Porting ALSA to RTLinux (using my DPI, which will provide a
95% standard kernel compatible API) would make sub sample accurate
lock *without flywheel* possible.
> 4) I think it would be a good thing to have a unique type Id for the streams
> with subtypes ? To be extremely generic, why not think about a tree /
> attributes scheme (in a string).
> (eg. : /audio/timedomain/?depth=16&voices=2&Fs=44100 or
> /video/mpeg4/?dummy=1&i_dunno_anything_about_mpeg4=5 for a MPEG4 video
> stream , MP3 streams would fit into /audio/freqdomain/...)
>
> A node would check about the input type by checking (ie with a regexp-style
> constraint string) the strings of the type to be checked . (ex :
> /audio/timedomain/AIFF/?depth=[16|8]&voices=* for a driver to match the
> audio streams of 16 or 8 bits and any number of voices)
There will be something like that, yes, but once set up, an actual
stream is nothing but a raw data stream. Somewhat like pipes, but
optimized for real time, high bandwidth streaming. (For example,
there are no read/write functions - the data is carried in buffers,
that may use shared memory to avoid copying. The latest idea is
controlling this using the event system.)
The init protocol seems to be going into the event system, as that
allows most of it to be used in the plugin API as well as the client
API.
> 5) just came in where it was about resampling : I dont't think (cubic or
> not) interpolation is a good idea. When it comes to interpolation, when you
> use a ploynomial interpolation, you make some nonlinear computation. Thus,
> you DISTORT your signal. A better way is to
> a) lowpass the signal with Fs (TO-BE Sampling Frequency)
> b) downsample the signal
>
> OR
>
> a) add some ZEROES between the samples to upsample the signal (there you
> don't change the spectrum of your signal, then
> b) lowpass the signal to cut the frequencies above the OLD sampling
> Frequency.
>
> Thus, you have NO LOSS (at all, I MEAN).
>
> There may be ways to make some rational quantification (eg. 4/7)
>
> Maybe you can have a look at the DSP FAQ which surely explains it better
> than I do. Of course it may be _very_ time consuming, but for a final mix,
> who cares ?
We deal mainly with real time, so we need a decent compromise as
well. (Actually, many users may never do off-line processing!)
We have discussed various resampling methods and come to the
conclusion that the only way is to try a few variants, and see which
one is fastest when using high enough accuracy to make it sound good.
> 6) Another Idea that I had is that we need several levels of libraries / API
> :
> - Common DSP things (resampling,
Plugin.
> file loading (AudioFile / Custom ...) ,
Engine (separate library), or an external client.
> fast mixing, FFTs (maybe by using FFTW)
Plugins.
> - A "node manager api" (like a window manager)
Yes.
> - Some example/basic plugins (and, first of all and most important, a
> DO-NOTHING plugin to make a TEMPLATE of simple plugins)
Well, there's lots of GPLed signal processing code out there. Just
start porting it! :-)
> - and, finally, a connexion with some kind of media players or higher
> levels.
Yep. A client API (similar to Benno's client/server proposal) with
an interface to the same event and stream system that the plugins
use. The idea is to keep these two versions of the API very similar,
so that you can turn plugins into clients and vice versa without too
much effort. It also simplifies integration, as you can easily
implement gateways for connecting things accross engines running
inside separate applications (=clients).
> 7) Do you support (or plan to support) multiple time references (eg
> different sound cards --> different clocks)? It can be handled with a
> general time reference (64 bit to be _very_ precise, and the fact that every
> clock should be able to convert itself to&from thsi ref. clock time, which
> wouldn't exist by itself, only in a reference basis between clocks)
There is one time base in a normal system; the master audio card.
Plugins inside engines will only see the local time, which is
measured in samples (or possibly fractions of samples for plugins
that want that), and relative to one of the external clients the
engine is connected to - in the normal case a sound card, or a
central mixing daemon.
Of course, you may run engines in different threads, synchronized to
different clients, and these will obviously have individual time
references. Whenever they connect to each other, they will have to be
aware of the fact that they may have different time bases.
<implementation>
Right now, I'd suggest using the resync points used for sensor
threads, when timestamping buffers sent to clients we're not synced
to: compare the most recent resync points to get the current offset
between the clients. Keep track of the *change* of the offset over
time to calculate the drift to use for resampling.
</implementation>
> 8) Of course I would like to help the project, but can you tell me : who is
> working on the project ?
Audiality: Me (if anyone at all)
The API: The ones who are discussing it on this list.
> Do you have a timeline ?
Well, "alpha spec ASAP", I guess... Anyway, IMO, getting it right is
slightly more important than getting it together and hacking away
ASAP. The alpha spec when it's released should be seen as very
unstable, as we'll probably find problems when people start using it.
Those problems will be *fixed* in a technically acceptable way; not
kept around as "holy legacy stuff" and worked around, as is often done
in the proprietary world.
> when did it start (The
> rt-linux thing, not the amiga ?)
The most interesting part that I've been involved in started with the
discussion on this list (whenever that was). A few weeks before
that, I got in touch with Benno and some other people, shortly after
publishing the Audiality/RTLinux site.
>From the AGC site ( http://www.angelfire.com/ar/agc/progress.html ):
"...the AGC Project is asleep, and probably will be for at least a
few months. (Jun 25, 1999) This is in favor for another project that
is more important to me, and has a better chance of becoming a
successful Free/Open Source project: The Audiality Project."
So, the Audiality/Linux site got up sometime around June 25...
However, that's not really relevant to the API design project. Some
of my ideas have developed over quite a few years, and nothing special
happened before this discussion took off.
> I could be a little help in programming, and maybe a better help in DSP (I
> passed all my exams of DSP...). My personnal goal is to make a Gnome/GTK+
> virtual studio & arranger studio, and i am lloking for an ausio core : I
> think I found what i need, and I would participate at this one instead of
> building my own.
That's exactly what we want with this; people getting involved in
making this *The* standard, rather than adding to the already way
too big pool of incompatible "standards".
So, point out any problems you find - your DSP knowledge may help
here. The new API is supposed to *work* in real life, so if it
doesn't, we just have to fix it somehow. Otherwise we'd be wasting
our time on this project.
//David
·A·U·D·I·A·L·I·T·Y· P r o f e s s i o n a l L i n u x A u d i o
- - ------------------------------------------------------------- - -
·Rock Solid David Olofson:
·Low Latency www.angelfire.com/or/audiality ·Audio Hacker
·Plug-Ins audiality_AT_swipnet.se ·Linux Advocate
·Open Source ·Singer/Composer
This archive was generated by hypermail 2b28 : pe maalis 10 2000 - 07:27:59 EST