Re: [linux-audio-dev] best option for audiovisual synchrony

Simon Barthelmé wrote:
> Hi all,
>
> We're in the process of porting an open source Matlab toolbox to Linux
> called the Psychtoolbox. It's a piece of software that's widely used
> within neuroscience and psychophysics, to display graphics and play
> sounds in experiments. It uses OpenGl for its graphics backend, and
> right now the sound support is just matlab's, which is really poor.
>
> A lot of us researchers are interested in audiovisual experiments, where
> we study how the brain combines auditory and visual information (for
> example in speech perception).
>
> Concretely speaking, that means we need to get as precise a timing and
> synchrony as we can possibly get. A typical experiment will go
> something like this :
> - display a flash and, at the same time, play a beep
> - wait for response
>
> The tricky bit is of course getting a flash that's totally synchronous
> with the beep. Absolute synchrony is not achievable without dedicated
> hardware, but we need to get an approximation that's within the few ms
> range.
>
> From what I've read it seems that Linux is a pretty good platform for
> that, since it has low latency/real time support.
> What I need to know is how I should go about implementing some kind of
> FlashAndBeep function, that guarantees that the sound starts playing as
> soon as the display buffer is flipped, with the shortest predictable
> delay possible.
>
> Is ALSA the way to go ? JACK maybe ? Anything else ?
>
> I'd love to read about any piece of advice you might have. Do ask me if
> you need more details.
>
>
> Thanks a lot,
>
> Simon Barthelmé
> PhD Student
> Laboratoire de Psychologie de la Perception
> CNRS/Université Paris V
>
> lpp.psycho.univ-paris5.fr
>
>
>

I don't know if you can do much about the video output. The delay from
sending the frame to it being displayed is quite short, as least short
enough not to worry movie playing in Linux.
I have done considerable work in xine to get audio lip sync good, and
being an ALSA kernel developer as well helped.
The best audio/video lip sync is achieved if you have all the content
ready some time before you wish to play it. I.e. A media file or some
program that knows in advance when it should play the video.
ALSA has a function call "snd_pcm_delay()"
This returns the number of frames (or samples) between the next one you
are about to write to ALSA, and the sound card hardware DAC playing the
sample. So, by using this function, one can accurately determine when
your sample will reach the speakers, and thus know exactly when to write
the samples you want to the buffer. You then keep the audio in sync with
a master system clock.
Separately, you keep the video in sync with a master system clock. The
end result is the audio and video are kept in perfect sync.

James
Received on Sat Oct 21 00:15:06 2006