Subject: Re: [linux-audio-dev] LAAGA API Proposal
From: Richard Guenther (rguenth_AT_tat.physik.uni-tuebingen.de)
Date: Wed Jun 13 2001 - 11:54:27 EEST
On Tue, 12 Jun 2001, Paul Davis wrote:
> >Before answering the following questions I like to tell about my view
> >of the typical usage pattern from the UI point of view. Lets suppose
>
> [ ... description elided ... ]
>
> yes, that more or less matches my notion of things closely enough that
> we clearly are aiming at the same general idea.
Ok, fine :)
> >I dont understand this sentence - so first for the bracketed stuff:
> >We dont have "automatic" mixing (I think the concept of automatic
> >mangling in any form is broken - the only useful think is buffer
> >replication (by reference or copying), but without change - this
> >allows to abstract the datatype from the LAAGA backend (remember
> >the guys speaking about midi...)).
>
> automatic mixing means that clients don't have to deal with the
> concept of "connections" or "buffers" or "mixing". if the user
> connects 6 outputs to a single input, the client doesn't have to
Not possible with my approach - there is no "input port" :) So I dont
have to handle that case (which doesnt mean you could wrap something
up that presents a LAAGA app with automatically mixed input).
> understand that it has to loop over them. In addition, we can support
It doesnt have to loop over them - well, it does have to loop over
all connections (buffers) it has explicitly created - I like this
notion, because there are no "invisible" side effects of having
multiple connections. Of course at the output side handling 1-to-n
transition is simple by implicitly copying/referencing inside the
LAAGA lib.
> ideas like universal connection gain controls without the clients
> knowing about it.
But again you assume you are dealing with f.i. float audio data -
I'd rather like to have the possibility to stream f.i. a video
stream along an audio stream (i.e. do _not_ assume anything about
the type of the data streamed inside LAAGA).
> >loop-until-no-more-buffers... well - you usually (for mono operation,
> >i.e. independend processing of connections) would do this inside
> >a thread (one per connection). For processing multiple connections
>
> yes, but in your model, this thread is also executing the core
> processing code for the client. in GLAME, you fetch the buffer(s), do
> any mixing necessary, work on the buffer, and then wait for the next
> buffer.
Remember that even with GLAME only those plugins need to handle mixing
that are willing to do so - for the other cases just use an explicit
mixing app/plugin.
Btw. your approach also does execute the core processing code inside
the callback, no? So again there is no difference between
while (1) {
sigwait();
process(data);
kill();
}
and
while (1) {
getbuffer();
/* inline processing code (or procedure call, if you like) */
putbuffer();
}
> you're doing this work in the client, right at the point where
> the client would like (i think) to be operating like a LADSPA plugin.
>
> still, i understand that you could wrap this. its really just a matter
> of which side of the IPC barrier you do this on, which i think is your
> point, right?
Yes - do the IPC barrier on the low-level'est place as possible (which
leaves us with the most powerful API) - be data type independend (for
the core), perhaps provide another "simple" API ontop of it which may
explicitly deal with audio data and provides a callback like operation.
> >at once (f.i. mixing) you need to be able to handle different sized
> >buffers from connections.
>
> different sized buffers? how so?
You have an app with two input ports (or connections) to two apps
which both do audio sampling putting out (whoops) different sized
buffers (which at least can happen anyway for the last buffer of
a stream - if there are "last" buffers - or do you pad those with
zeroes?). Again for "simple" apps this is not an issue as they
deal with a single connection only.
> >Now a general comment: You absolutely can wrap an API like yours
> >(with callbacks) around the above concept - but not the other way
> >around.
>
> Let me think about this.
You can always wrap "powerful, simple" -> "special", but not the
other way around. Let me give an example: for a visualization app
you dont want to process every buffer at any cost - you just want
(if you happen to get a timeslice) to get the most recent buffer
in a non-blocking way (so you're running in "network sniffer" mode).
So in the app you'd rather want to do processing inside f.i. a gtk_idle()
function which seems not possible with your approach (the signal
will probably arrive during some blocking read/write inside
GUI code and doesnt trigger again during the call of gtk_idle()
where you can't afford to block possibly stalling the gui).
Q: What happens, if I "ignore" a kill()? The network does stall,
doesnt it (even if I dont produce output another app depends
on)? So you have big problems avoiding signal lossage - as
you just use sigwait() and it is unspecified what happens for
other code, with blocking
sig ---> read()/write() [or without]
sigwait();
(see http://www.opengroup.org/onlinepubs/7908799/toc.htm)
You also interfere with non-RT signal capable kernels and/or
old libc pthread implementations which use SIGUSR1/2 for
internal communication.
> > And for the simple cases (with your API, an app with two
> >input ports (stereo) will receive buffers on those in sync? I.e.
> >inside one callback? I dont see how you handle this at all)
>
> of course they are in sync. the engine has a sorted execution list; a
> client expecting to receive data is only executed after its data
> sources have already executed; it gets called once via its process()
> callback, and can get the memory areas associated with both buffers at
> once to use within that callback.
Ok.
> >Have I mentioned that in my above example we dont have an engine at all?
> >(you could call the audioio app the engine, though)
>
> Ok, so in your model, there is no single central point where the
> "signal" to initiate the processing the graph originates. Instead,
> that "signal" could (theoretically) originate from anywhere. In the
> real world, it will occur once some component has a buffer ready to be
> delivered, such as the audioio app, which will drive all those
> connected to it. These in turn will drive their buffers through to the
> final destinations.
>
> Did I get this right?
Yes.
> I can see some potential problems with this approach, but first I want
> to make sure I understand it correctly.
>
> >> my preference for a model where once you get the address of the single
> >> memory region associated with the port, its just a chunk of memory as
> >> it would be in a very simple plugin system (e.g. LADSPA). that is, its
> >
> >Hey wait - we dont want to do LADSPA2, do we? Interprocess communication
> >is nowhere like connecting LADSPA plugins (though its possible, but you
> >dont want that).
>
> Well, actually, I do want that :) I just want to support typed ports
> and out-of-process clients as well.
Umm - for internal operation you dont want to use LAAGA - instead you
want to use the specialized LADSPA stuff - else you could just do
an IPC implementation of the LADSPA API.
> >I dont think explicit buffer handling is complex at all. It just makes
> >things like echo/delay/feedback possible without doing handcrafted
> >ringbuffers (and as such avoids unnecessary copies).
>
> i may be dumb, but my understanding is that:
>
> 1) you can't (correctly) implement any feedback-requiring DSP with a buffer
> size larger than 1 frame in *any* system
?? We (GLAME) do implement (correctly? whats correctly??) feedback
requiring filters both "internally" (by using a linked list of
buffers -- see the echo.c filter) and "externally" by constructing
a network with feedback (see the echo2.scm filter). It works like
a charm (you _need_ async. processing for such thing to work) -
but I dont know if its still "incorrect".
> 2) the model you've suggested doesn't seem to me to solve this
> any better than the one i have offered.
You have sync. processing, so the engine has to know if apps "extend"
or "shorten" the audio stream [delay] and it has to explicitly handle
feedback (which you get for free with an async. model) as in:
-->--- mix -------------->--------------\---->---
/ |
\--<--delay---<---volume_adjust--<---/
(which happens to be the layout of our echo2 filter network)
> > If the graph
> >is async. driven - but by audio I/O - async. and sync. operation are
> >the _same_
>
> OK, point taken.
Ah, nice - so we dont argue about sync./async. operation anymore? :)
Richard.
-- Richard Guenther <richard.guenther_AT_uni-tuebingen.de> WWW: http://www.tat.physik.uni-tuebingen.de/~rguenth/ The GLAME Project: http://www.glame.de/
This archive was generated by hypermail 2b28 : Wed Jun 13 2001 - 13:13:31 EEST