Subject: [linux-audio-dev] Re: sfront 0.65 09/07/00
From: John Lazzaro (lazzaro_AT_CS.Berkeley.EDU)
Date: Thu Sep 21 2000 - 23:08:49 EEST
> I may be totally on the wrong track here, but I'm interested
> in the possibility of using SAOL to build complex
> high-quality dsp applications without having to muck around
> in C. :)
Cool, here are some ideas ... I guess the most-significant-bit of my
response is that I see myself as setting up the sfront infrastructure
for these sorts of additions, so that people can write them and
contribute the source back w/o having to learn sfront internals. So,
most of my responses will be in the form "here's what to read to learn
how to add this feature yourself :-)", along with the "the
infrastructure isn't there yet but I'm working on it" reply and the
ever-popular "that's MPEG's problem not mine" :-).
Basically, the existing infrastructure I'll be referring to will
either be
"you can write a control driver to do this" which means reading:
http://www.cs.berkeley.edu/~lazzaro/sa/sfman/devel/cdriver/intro/index.html
"you can write an audio driver to do this" which means reading:
http://www.cs.berkeley.edu/~lazzaro/sa/sfman/devel/adriver/index.html
or occassionally, references to places in the sfront source itself.
The "infrastructure under development" I refer to is letting an
sfront-created sa.c engine send and receive multiple RTP-over-UDP
streams of SA_access_units. SA_access_units can send MIDI or SASL --
and since SASL commands include the table command, its possible to
send packets of audio data into the engine. In fact, the SDF-to-MP4
converter uses this trick to send SDF data into sfront, where SAOL
commands turn it into audio. I'm focusing on the RTP infrastructure
because it supports peer-to-peer Internet applications well ...
Oh, RTP is:
http://www.cs.columbia.edu/~hgs/rtp/
With that intro ...
> 1) As far as I can tell, "sample" and the input bus are the
> only ways to get audio from disk files into SAOL.
The most general way to do what you want is to send SA_access_units
into the sa.c engine, that use the table and control SASL commands
together to make blocks of sample data visible to the SAOL code. Table
commands fill up an existing global or "future" table with data, and
you use control commands as semaphores to signal new data input to
your SAOL code.
Eric Scheirer used this trick in his SDF to SAOL converter, and it
works quite well. If bandwidth isn't a problem, you can use raw "data"
wavetable generators, if you want to be craftier you may be able to
make one of the other wavetable generators get the data into the
engine in a more compressed way. This is all 32-bit float data, so
that issue is solved.
How to do this? Today, it means writing your own control driver,
its also one of the classic apps I'm writing the RTP interface
to handle. Note that today, you only get 1 control driver in
sfront -- I'm hoping the RTP interface will be nice enough that
multiple-driver apps get written that way, but if not there's
nothing fundamental preventing sfront supporting multiple -cin's,
although its a major infrastructure project, and a less general
one than the RTP one (which is why its on the back burner).
Other, less satisfying ways to do this would be:
[1] Use a wide input_bus, write your own audio driver to handle
the wide input bus, and fake all the asynchronous stop-start
activity in your driver, by sending zeros when there isn't any
data, ect. Note the audio driver interface handles 32-bit floats
and 16-bit shorts, so that issue is handled.
[2] Use vanilla sample generator commands, but rewrite the
C code sfront generates for sample generators to handle
very large files well -- you'd be detecting the size of
the data file early on, and if its "small" you'd load it
in a dumb way, and if its "big" you'd be smarter about
it. One of the nice things about SAOL is that tables can
only be handled by one of a small number of table opcodes
directly -- so you'd teach the code-generators of all
of those opcodes how to handle both "regular" and "king-size"
tables, and then you'd be covered (note you can tell the
difference at sfront runtime). See my answer to the next
question for more details. You'd handle the "asynchronous"
part by having the table generators in SAOL instrs, and
controlling when you launched the instr.
> 2) Will there ever be support for sample formats other than
> 16-bit int? In particular, 32-bit float and 24-bit int would
> be very nice. Also, the mono-only restriction on the sample
> opcode is a bit irritating!
All but "mono-only" is in sfront's control -- sfront 0.6{5,6,7} is
much saner when it comes to WAV/AIF input, it reads the sample file
in:
function sampleconstcheck() in wtconst.c,
and grabs all of the "control information" out of it,
such as sample rate and loop points. Come codegen time,
it spits out a small bit of code that (conceptually)
fseeks's to the data block and reads it in, see:
function samplefilecode() in wtparse.c
to see the actual code generation. In addition, sample
files are read as part of the process of making an .mp4
binary file from ASCII SAOL and SASL, see:
function nextsamplewrite() in mp4write.c
To handle (say) the 32-bit float WAV file, these three
sections above would have to be changed. Technically,
since the "sample" wavetable generator is not part of
the MPEG-4 standard itself (only the binary format of
the .mp4 file that holds the sample data), you could
add an optional parameter to "sample", that was 0
for "read channel 1 of the WAV file", 1 for "read
channel 2 of the WAV file", ect, to make the mono-only
issue slightly less irritating, but there's no
standards-compliant way to do what you really want,
which is stereo wavetables as a datatype in the
language itself ... you could imagine writing some
core opcodes to simplify dealing with them, but
they'll feel like complex-number function libraries
in C feel like (as opposed to FORTRAN's built-in
complex type).
> 3) One thing I like very much about csound on linux is the
> -L command-line flag, which lets me pipe ASCII score events
> to csound's standard input from any program I like. [...]
You're right, there's no real-time SASL input control driver
for sfront right now -- only linmidi for MIDI input. You
can write a control driver to do this -- in fact, if you
look at the gliss example, and look at the gliss control
driver in sfront/src/lib/csys/gliss.c, you'll see that the
up-glisses are MIDI commands and the down-glisses are SASL
commands, so that's a simple example to see SASL control
driving in action. The -cin fstr driver also handles SASL
control input -- it opens up a -mp4 file to the start of
the SA_access_unit block and starts parsing. Either of
these could be a starting point for a control driver to
do what you want.
> 4) Likewise, I sometimes find it useful to connect MIDI
> applications via stdin and stdout. (This would probably be
> easier to implement.)
Stdin is easy -- you might try cloning the linmidi code, and replacing
"/dev/midi" with "stdin", then start examining the code to see what
else would have to change (perhaps not that much). Stdout is an
unsolved infrastructure problem at the moment, because what you really
want is "trace" functionality in the sa.c engine, which makes calls to
a -cout API and/or injects data into a RTP SA_access_unit stream, when
SAOL and SASL events happen. I'm interesting in "doing this right",
which means taking the time necessary to find the right semantics that
doesn't require adding SAOL or SASL commands (that's MPEG's domain,
not mine), so don't expect to be able to do this for a while.
> 5) Would it be possible to provide an option to generate
> LADSPA plugins with sfront? That would be really cool!
I don't know enough about LADSPA to say how easy or hard this would be
-- maybe the right meta-comment to make is that I can easily imagine
sfront supporting command-line directives for replacing the default
"sfront/src/lib/csrc/preamble.c" (the top of an sa.c file) and
"sfront/src/lib/csrc/runtimef.c" (the main() at the bottom of an sa.c
file, f is for the float interface to audio drivers) code with stubs
needed to interface into LADSPA -- if this is all it takes to make a
plugin happen, then it shouldn't require much infrastructure on my
end. If it turns out that SAOL-program-specific code generation needs
to change, though, things become harder ...
> 6) Will sfront support multi-channel soundcards? ALSA?
The audio driver API can handle this -- it can handle arbitrary number
of channels and it can handle 32-bit float in and out. What's needed
is an ALSA driver for sfront, written by someone who actually has the
high-end hardware ALSA supports so well, so that ALSA users no longer
need to use the OSS -aout linux option. See:
http://www.cs.berkeley.edu/~lazzaro/sa/sfman/devel/adriver/index.html
for info on writing audio drivers (hoping to improve this part of the
documentation soon, though).
> Do you know if the saol mailing list is active?
> I sent a message to saol-users-request_AT_media.mit.edu
> but I haven't received any reply.
People send messages to it occassionally (much more often to saol-dev)
so it works for existing users, I don't know if there's issues with
adding/deleting users ... sometime the Media Lab mail servers take
a long time to reply, maybe you've received something by now.
--jl
-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------
This archive was generated by hypermail 2b28 : Fri Sep 22 2000 - 00:28:17 EEST