Subject: Re: [linux-audio-dev] LAAGA API Proposal
From: Richard Guenther (rguenth_AT_tat.physik.uni-tuebingen.de)
Date: Wed Jun 13 2001 - 16:56:45 EEST
On Wed, 13 Jun 2001, Paul Davis wrote:
> >> automatic mixing means that clients don't have to deal with the
> >> concept of "connections" or "buffers" or "mixing". if the user
> >> connects 6 outputs to a single input, the client doesn't have to
> >
> >Not possible with my approach - there is no "input port" :) So I dont
> >have to handle that case (which doesnt mean you could wrap something
> >up that presents a LAAGA app with automatically mixed input).
>
> Well, I consider the concept of "unlimited numbers of signals being
> delivered to a single location" rather important, since it is a direct
> analog for what we do with a bus in the analog world. If we start to
> require predefined numbers of <something> in order to do mixing, its
> hard for me to see how to implement a bus.
>
> However, if I understand your model well enough, you can still make N
> connections - you just have to call getBuffer() on each one, then
> mixdown the output, then queueBuffer() with the result. True?
Yes - there is just no single point (no input ports) where we can do
automatic mixing.
> >It doesnt have to loop over them - well, it does have to loop over
> >all connections (buffers) it has explicitly created - I like this
> >notion, because there are no "invisible" side effects of having
> >multiple connections.
>
> The effect is never invisible. As you note later, its really just a
> question of where the code lives to do the mixing.
>
> > Of course at the output side handling 1-to-n
> >transition is simple by implicitly copying/referencing inside the
> >LAAGA lib.
>
> Of course (assuming either no builtin-gain concept, or unity gain).
>
> >> ideas like universal connection gain controls without the clients
> >> knowing about it.
> >
> >But again you assume you are dealing with f.i. float audio data -
> >I'd rather like to have the possibility to stream f.i. a video
> >stream along an audio stream (i.e. do _not_ assume anything about
> >the type of the data streamed inside LAAGA).
>
> No, I'm only assuming that one of the types of stream is an audio
> stream and that we enforce typing when connecting ports. Nothing
> in the client API says that you can or cannot connect multiple (say)
> video streams to a single point. If someone implemented a LAAGA engine
> that understood how to mix video streams, then it could implement a
> "bus" like model for video. If not, then any attempt to make double
> connections with video will fail.
Ok, with my approach there is no engine, so any automatic stuff has
to be done in a LAAGA library which for different types (if it cares
about types at all) leads to versioning problems.
> In my initial implementation, audio is the only "builtin" type for a
> port, but any type is allowed to exist.
>
> >Remember that even with GLAME only those plugins need to handle mixing
> >that are willing to do so - for the other cases just use an explicit
> >mixing app/plugin.
>
> How does a 3rd party behave (both visibly and code-wise) when a user
> tries to make a second connection using an object that is already
> connected and doesn't do mixing?
The connection is refused. So the user has to insert a mixing plugin
inbetween (if the desired effect was mixing - which is intuitive, but
obviously not the only possible effect).
> >> >at once (f.i. mixing) you need to be able to handle different sized
> >> >buffers from connections.
> >>
> >> different sized buffers? how so?
> >
> >You have an app with two input ports (or connections) to two apps
> >which both do audio sampling putting out (whoops) different sized
> >buffers (which at least can happen anyway for the last buffer of
> >a stream - if there are "last" buffers - or do you pad those with
> >zeroes?).
>
> In a synchronously running system (whether or not its underlying
> architecture may also support async operation), this can't
> happen. Things are driven by the "an audio interface", and the amount
> of data type of type T being passed around must be the same
> everywhere, otherwise things are no longer in sync. Obviously, we can
> pass X bytes of MIDI and Y bytes of audio, but everywhere that does
> audio (of a given type) must be moving Y bytes.
Ah, ok - this simplifies (a little bit) buffer handling and processing,
but makes feedback with fifo size != fragment size impossible (you said
that already, if I understood right).
> >You can always wrap "powerful, simple" -> "special", but not the
> >other way around. Let me give an example: for a visualization app
> >you dont want to process every buffer at any cost - you just want
> >(if you happen to get a timeslice) to get the most recent buffer
> >in a non-blocking way (so you're running in "network sniffer" mode).
> >So in the app you'd rather want to do processing inside f.i. a gtk_idle()
> >function which seems not possible with your approach (the signal
> >will probably arrive during some blocking read/write inside
> >GUI code and doesnt trigger again during the call of gtk_idle()
> >where you can't afford to block possibly stalling the gui).
>
> I am really very opposed to this model. If you are going to allow
> something like this to be done, I don't believe that you can support
> low latency operation in general. In your model, I think that
> queueBuffer() is responsible for "driving" the next stage of the
> graph.
Yes, but obviously only if there are dependencies on the output.
> It therefore has to send a "signal" of some kind to the object
> that is on the other end of the connection. If this "signal" can be
> ignored, there's no way to complete the graph execution. Moreover, the
> code that handles the "signal" needs to execute with RT-like
> characteristics (IPC notwithstanding). So you cannot possibly handle
> the receipt of whatever "signal" queueBuffer() "sends" from within
> gtk_idle(). You can do internal "queuing", and only draw within
> gtk_idle(), but you can't handle the graph execution signal from there.
Well, queueBuffer() does what it is called - it just queues the buffer
to the recipients buffer lists. Any recipient blocking on its (empty)
buffer list gets woken up (easy to implement with unix pipes or the
like, writes for queue(), blocking (or nonblocking for poll/select
like operation) reads for get()).
So there is no problem with an app which does just "listen" to process
data within gtk_idle() - obviously the apps buffer queue may fill up,
but hey - thats what you want (because of latency).
> >Q: What happens, if I "ignore" a kill()? The network does stall,
> > doesnt it (even if I dont produce output another app depends
> > on)? So you have big problems avoiding signal lossage - as
> > you just use sigwait() and it is unspecified what happens for
> > other code, with blocking
> >sig ---> read()/write() [or without]
> > sigwait();
>
> well, now you begin to see the difference between sync and async
> operation.
>
> in my model, what you describe *cannot* happen. there is a dedicated
> thread that executes only when it receives a signal (of some kind)
> from the previous stage of the graph. while it executes, no other
> signals will be delivered (or certainly not to it). once its done, it
> delivers another signal to the next stage of the graph, and goes back
> to sleep.
>
> this is what bothers about your "async" model. it sounds as if other
> components could potentially use queueBuffer() to signal a component
> at any time, making it very hard to write the code so that it works in
> a low latency situation. in my system, whenever a component is
> running, it already knows that all of its buffers are ready and that
> it will not interrupted or resignalled or whatever during its
> execution.
Humm - so you cant handle "independend" graphs without being forced
to sync them? I.e. no SMP benefit at all?
> > You also interfere with non-RT signal capable kernels and/or
> > old libc pthread implementations which use SIGUSR1/2 for
> > internal communication.
>
> yes, i know about that. i've talked to the pthread author about this,
> and he considers that version to be broken. kernel 2.2 and above
> support the "RT" signal set, so I don't consider this much of a
> problem.
Err - so you're Linux only and dont support BSD or other weird
architectures. This is another reason I dont like signals - signals
and threads dont mix portably, with read()/write()/select() you have
maximum portability (even NT might support such style of operation).
But of course using signals is implementation dependant and no
requirement for the API.
> >> >I dont think explicit buffer handling is complex at all. It just makes
> >> >things like echo/delay/feedback possible without doing handcrafted
> >> >ringbuffers (and as such avoids unnecessary copies).
> >>
> >> i may be dumb, but my understanding is that:
> >>
> >> 1) you can't (correctly) implement any feedback-requiring DSP with a buff
> >er
> >> size larger than 1 frame in *any* system
> >
> >?? We (GLAME) do implement (correctly? whats correctly??) feedback
>
> Maarten had some stuff about this when he wrote/released tapir
> (sp?). I can't remember the exact details, but I do have some memory
> of a good explanation of why you need sample-by-sample processing to
> do this correctly.
You obviously have latency of at least the buffer size - but you have
that always (but that wasnt your point?)
> >requiring filters both "internally" (by using a linked list of
> >buffers -- see the echo.c filter) and "externally" by constructing
> >a network with feedback (see the echo2.scm filter). It works like
> >a charm (you _need_ async. processing for such thing to work) -
> >but I dont know if its still "incorrect".
>
> you don't need async for this to work. if there is a feedback loop,
> there is no correct order for the graph execution, so you merely
> need a guarantee of a particular order for as the loop exists.
There is a correct order - for the echo example, the first plugin
to execute is the delay plugin which needs to put out a set of zeros
to be able to start processing in the other nodes. So for sync.
operation you somehow magically need to detect that delay can produce
output without having input first.
> >> 2) the model you've suggested doesn't seem to me to solve this
> >> any better than the one i have offered.
> >
> >You have sync. processing, so the engine has to know if apps "extend"
> >or "shorten" the audio stream [delay] and it has to explicitly handle
> >feedback (which you get for free with an async. model) as in:
>
> No. You are still thinking about feeding fixed length audio data
> through the system, such as when processing a file.
>
> LAAGA doesn't do this - its a free-running system into which audio of
> fixed length can be injected, but it continues to run before, during
> and after that particular length is done. We don't care about the
> "extension" effect of a delay line, though the application containing
> the delay line probably does. (*)
>
> --p
Ok, we seem to know what both approaches do and what advantages and
disadvantages are (but we dont agree on them). So I think we either
need input from some other guy or we can stop discussion.
Perhaps I have time to do an implementation of my API, probably calling
it something different than LAAGA - ASCA (Application Stream Communication
API). Competition is always good.
Richard.
> (*) this does raise one other nasty problem with the current prototype
> that i have. if you have short-lived clients coming and going, it
> seems hard to detect their departure "on time". we do detect it,
> but only because the watchdog timer goes off, and then we can
> clean up. it would be much nicer to be able to detect this
> on-time, but it appears that kill(2) does not return an error when
> a process is "dying". this isn't fixed by using read/write either,
> since it appears that poll(2) doesn't return an error when a
> socket is "closing".
>
> as i said, this only really affects short-lived clients that
> come and go, and i don't imagine that in a real situation
> this would really be the case.
>
-- Richard Guenther <richard.guenther_AT_uni-tuebingen.de> WWW: http://www.tat.physik.uni-tuebingen.de/~rguenth/ The GLAME Project: http://www.glame.de/
This archive was generated by hypermail 2b28 : Wed Jun 13 2001 - 18:27:16 EEST