Subject: Re: [linux-audio-dev] LAAGA API Proposal
From: Paul Davis (pbd_AT_Op.Net)
Date: Wed Jun 13 2001 - 16:38:58 EEST
>> automatic mixing means that clients don't have to deal with the
>> concept of "connections" or "buffers" or "mixing". if the user
>> connects 6 outputs to a single input, the client doesn't have to
>
>Not possible with my approach - there is no "input port" :) So I dont
>have to handle that case (which doesnt mean you could wrap something
>up that presents a LAAGA app with automatically mixed input).
Well, I consider the concept of "unlimited numbers of signals being
delivered to a single location" rather important, since it is a direct
analog for what we do with a bus in the analog world. If we start to
require predefined numbers of <something> in order to do mixing, its
hard for me to see how to implement a bus.
However, if I understand your model well enough, you can still make N
connections - you just have to call getBuffer() on each one, then
mixdown the output, then queueBuffer() with the result. True?
>It doesnt have to loop over them - well, it does have to loop over
>all connections (buffers) it has explicitly created - I like this
>notion, because there are no "invisible" side effects of having
>multiple connections.
The effect is never invisible. As you note later, its really just a
question of where the code lives to do the mixing.
> Of course at the output side handling 1-to-n
>transition is simple by implicitly copying/referencing inside the
>LAAGA lib.
Of course (assuming either no builtin-gain concept, or unity gain).
>> ideas like universal connection gain controls without the clients
>> knowing about it.
>
>But again you assume you are dealing with f.i. float audio data -
>I'd rather like to have the possibility to stream f.i. a video
>stream along an audio stream (i.e. do _not_ assume anything about
>the type of the data streamed inside LAAGA).
No, I'm only assuming that one of the types of stream is an audio
stream and that we enforce typing when connecting ports. Nothing
in the client API says that you can or cannot connect multiple (say)
video streams to a single point. If someone implemented a LAAGA engine
that understood how to mix video streams, then it could implement a
"bus" like model for video. If not, then any attempt to make double
connections with video will fail.
In my initial implementation, audio is the only "builtin" type for a
port, but any type is allowed to exist.
>Remember that even with GLAME only those plugins need to handle mixing
>that are willing to do so - for the other cases just use an explicit
>mixing app/plugin.
How does a 3rd party behave (both visibly and code-wise) when a user
tries to make a second connection using an object that is already
connected and doesn't do mixing?
>> >at once (f.i. mixing) you need to be able to handle different sized
>> >buffers from connections.
>>
>> different sized buffers? how so?
>
>You have an app with two input ports (or connections) to two apps
>which both do audio sampling putting out (whoops) different sized
>buffers (which at least can happen anyway for the last buffer of
>a stream - if there are "last" buffers - or do you pad those with
>zeroes?).
In a synchronously running system (whether or not its underlying
architecture may also support async operation), this can't
happen. Things are driven by the "an audio interface", and the amount
of data type of type T being passed around must be the same
everywhere, otherwise things are no longer in sync. Obviously, we can
pass X bytes of MIDI and Y bytes of audio, but everywhere that does
audio (of a given type) must be moving Y bytes.
>You can always wrap "powerful, simple" -> "special", but not the
>other way around. Let me give an example: for a visualization app
>you dont want to process every buffer at any cost - you just want
>(if you happen to get a timeslice) to get the most recent buffer
>in a non-blocking way (so you're running in "network sniffer" mode).
>So in the app you'd rather want to do processing inside f.i. a gtk_idle()
>function which seems not possible with your approach (the signal
>will probably arrive during some blocking read/write inside
>GUI code and doesnt trigger again during the call of gtk_idle()
>where you can't afford to block possibly stalling the gui).
I am really very opposed to this model. If you are going to allow
something like this to be done, I don't believe that you can support
low latency operation in general. In your model, I think that
queueBuffer() is responsible for "driving" the next stage of the
graph. It therefore has to send a "signal" of some kind to the object
that is on the other end of the connection. If this "signal" can be
ignored, there's no way to complete the graph execution. Moreover, the
code that handles the "signal" needs to execute with RT-like
characteristics (IPC notwithstanding). So you cannot possibly handle
the receipt of whatever "signal" queueBuffer() "sends" from within
gtk_idle(). You can do internal "queuing", and only draw within
gtk_idle(), but you can't handle the graph execution signal from there.
>Q: What happens, if I "ignore" a kill()? The network does stall,
> doesnt it (even if I dont produce output another app depends
> on)? So you have big problems avoiding signal lossage - as
> you just use sigwait() and it is unspecified what happens for
> other code, with blocking
>sig ---> read()/write() [or without]
> sigwait();
well, now you begin to see the difference between sync and async
operation.
in my model, what you describe *cannot* happen. there is a dedicated
thread that executes only when it receives a signal (of some kind)
from the previous stage of the graph. while it executes, no other
signals will be delivered (or certainly not to it). once its done, it
delivers another signal to the next stage of the graph, and goes back
to sleep.
this is what bothers about your "async" model. it sounds as if other
components could potentially use queueBuffer() to signal a component
at any time, making it very hard to write the code so that it works in
a low latency situation. in my system, whenever a component is
running, it already knows that all of its buffers are ready and that
it will not interrupted or resignalled or whatever during its
execution.
> You also interfere with non-RT signal capable kernels and/or
> old libc pthread implementations which use SIGUSR1/2 for
> internal communication.
yes, i know about that. i've talked to the pthread author about this,
and he considers that version to be broken. kernel 2.2 and above
support the "RT" signal set, so I don't consider this much of a
problem.
>> >I dont think explicit buffer handling is complex at all. It just makes
>> >things like echo/delay/feedback possible without doing handcrafted
>> >ringbuffers (and as such avoids unnecessary copies).
>>
>> i may be dumb, but my understanding is that:
>>
>> 1) you can't (correctly) implement any feedback-requiring DSP with a buff
>er
>> size larger than 1 frame in *any* system
>
>?? We (GLAME) do implement (correctly? whats correctly??) feedback
Maarten had some stuff about this when he wrote/released tapir
(sp?). I can't remember the exact details, but I do have some memory
of a good explanation of why you need sample-by-sample processing to
do this correctly.
>requiring filters both "internally" (by using a linked list of
>buffers -- see the echo.c filter) and "externally" by constructing
>a network with feedback (see the echo2.scm filter). It works like
>a charm (you _need_ async. processing for such thing to work) -
>but I dont know if its still "incorrect".
you don't need async for this to work. if there is a feedback loop,
there is no correct order for the graph execution, so you merely
need a guarantee of a particular order for as the loop exists.
>> 2) the model you've suggested doesn't seem to me to solve this
>> any better than the one i have offered.
>
>You have sync. processing, so the engine has to know if apps "extend"
>or "shorten" the audio stream [delay] and it has to explicitly handle
>feedback (which you get for free with an async. model) as in:
No. You are still thinking about feeding fixed length audio data
through the system, such as when processing a file.
LAAGA doesn't do this - its a free-running system into which audio of
fixed length can be injected, but it continues to run before, during
and after that particular length is done. We don't care about the
"extension" effect of a delay line, though the application containing
the delay line probably does. (*)
--p
(*) this does raise one other nasty problem with the current prototype
that i have. if you have short-lived clients coming and going, it
seems hard to detect their departure "on time". we do detect it,
but only because the watchdog timer goes off, and then we can
clean up. it would be much nicer to be able to detect this
on-time, but it appears that kill(2) does not return an error when
a process is "dying". this isn't fixed by using read/write either,
since it appears that poll(2) doesn't return an error when a
socket is "closing".
as i said, this only really affects short-lived clients that
come and go, and i don't imagine that in a real situation
this would really be the case.
This archive was generated by hypermail 2b28 : Wed Jun 13 2001 - 18:27:59 EEST