Re: [LAD] strategies for mixing streams of differing channel counts

On Sat, May 02, 2015 at 08:30:50AM +0000, dave@email-addr-hidden wrote:

> Could I get some ideas and advice on the subject of reconciling two
> streams with different channel layouts?

Hold on, weren't you the guy who couldn't mix two audio streams into one
three days ago, and now you want to address surround?

How about taking a step back and getting the basics, first? Correct
surround sound is a delicate subject, and entire companies have been
founded to get it right.

If you really want to, read some basics about positioning sound sources,
maybe start with the many papers on spatial audio from the past Linux
Audio Conferences. It won't get you anywhere near, but it will get you
started. Expect some math to be involved in this.

> with both channels carrying the same data. Mixing a one or two
> channel stream with 5.1 or 7.1 also seems fairly clear: the audio
> goes to the front left and right channels. Is this line of thinking
> solid?

No. Upmixing from stereo to any kind of surround can be arbitrarily
complex. If you want it centered, send it to front-left, front-right,
rear-left and rear-right with different gains depending on your desired
position. Sum it to mono and apply an LPF with an appropriate cut-off for
your .1 bass channel.

> I understand that the chances of someone writing a game that would
> go beyond stereo are rather slim,

Quite the contrary, the industry is already doing it:

   http://www.iosono-sound.com/game-audio/

> I'm also considering extending the results of my work into a
> general-purpose mixer library.

Have you already figured out what mix buffers are and how you keep a
single stereo stream open that is fed from multiple sources? To save you
some time, here's how to do it properly:

 * keep the stream open all the time - send zeroes if you have nothing
   to output

 * don't rely on external software (like pulse) for mixing multiple
   streams. That said, send a single stereo stream from your
   application.

 * use float32 as the internal representation, with audio normalised to
   -1.0 .. 1.0.

 * mixing is adding two floats:
   float a, b, mix; mix = a + b;

 * volume is multiplying with a scalar (usually 0..1, occasionally
   larger): float in, out, gain; out = in * gain;

 * you might find non-interleaved channels easier to handle:
   float left[512], right[512] instead of float stereo[1024]

 * ask your libs (like libsndfile) to return you floats, then sum the
   individual buffers and hand the result to the output stream. If need
   be, lower the volume as a final step.

 * use power-of-two block sizes. Not strictly necessary but convenient.

 * Keep an eye on SIMD

All in one:

float lout[512], rout[512];
float output_gain = 0.5;
for i=0; i < bufszie; i++ {
  lout[i] = lsrc1[i] + lsrc2[i] ... // potentially another for loop
  rout[i] = rsrc1[i] + rsrc1[i] ...
  lout[i] *= output_gain; // or fold into line about
  rout[i] *= output_gain;
}

If you have many sources, you will ultimately write a matrix mixer. You
have N inputs an M outputs. Your NxM matrix then holds the gain values
that express how much of input n should go to output m. Zero means
nothing, one is "all". M will most likely be two (stereo).

If you're striving for perfection, mind cache efficiency and the order
in which you nest the loops. The float buffers are contiguous and hence
benefit from hardware prefetching if you traverse them linearly.

Cheers and good luck

-- 
mail: adi@email-addr-hidden  	http://adi.thur.de	PGP/GPG: key via keyserver
_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/listinfo/linux-audio-dev