Subject: Re: [linux-audio-dev] LADSPA 64bit FP support ?
From: David Olofson (david_AT_gardena.net)
Date: Thu Mar 30 2000 - 04:49:30 EEST
On Sun, 26 Mar 2000, Karl W. MacMillan wrote:
> > > What type of data can you not transfer between plugins as vector or scalar
> > > floats that the data types that you added will allow?
> >
> > In theory you cold probably pass anything that way, but if you're
> > going to use MMX, deal with high frequency data (like from digital
> > oscilloscope cards) or video data, it's not very suitable.
>
> This is a philosophical road we probably don't want to go down, but I
> think that assembler is better written by compilers
I agree these days (used to be an asm die-hard, but that was when the
difference was to big to be acceptable for many real life
applications), but compilers don't generate SIMD instructions from
normal code yet. It's either asm or macros.
> and I don't want to
> write for specific processors (there is always something better just
> around the corner anyway).
Yes, but array processing means speedups in the range 200% - 800%, so
you'll have to wait a long time... And then, there'll probably be at
least 256 bit busses and some 1600% boost or more for SIMD.
Also, some people have to get their real time processing done within
a sane budget. Not everyone can afford a dual Alpha workstation as an
easier-to-program alternative to a UP P-III or G4. (The difference
may well be that big!)
> Don't the new instructions for the PIIIs that
> magically accelerate the internet do floating point anyway?
Yes, but it's still SIMD with special op-codes; not magic. How to make
an optimizer unroll loops into efficient code using packed data types
and SIMD instructions? I'm sure it's possible to do in theory, but I
seems to me that the complexity suggests AI solutions.
The alternative is improving the CPU pipelines and parallel
processing capabilities far beyond the levels of any current CPU.
This isn't exactly trivial either, as it takes *lots* of chip space
and still has to keep track of result dependencies atexecution
time... (SIMD has no dependencies between fields of packed data,
which makes it very trivial to implement, compared to run time
optimizing solutions. It's just lots of adders, multiplication arrays
etc doing the same operations at the same time.)
Can't say that I know which way is easiest, but in general, software
is cheaper for large volumes, so I would guess that the compilers
will eventually be capable of generating SIMD code from "real" source
code.
> Don't know
> enough to comment on the other two. My only thought is that it would be a
> shame to require extra code in 99 percent of the plugins just to
Of course. Any extra code with the possible exception of one or two
lines of init time code is banned, more or less. Slowing
implementations down is failing to implement the idea, as the whole
point is to enable performance increase and more flexibility.
> facilitate a few plugins that are probably going to require special hosts,
There's no requirement for special hosts if the API is designed for
multiple datatypes in a proper way.
> etc. anyway. You never know, though, running audio through plugins
> designed for video might be fun . . .
Controlling video effects from audio analyzer plugins is essential,
rather than fun. Don't know about passing raw data... *hehe*
Mixing modules written for instrumentation, control, audio etc
sounds very interesting, though, and they all use quite similar data
formats.
> > One could argue that this API will only be used for "power consumer"
> > through high end audio, and that plugins using different data types
> > will never run under the same host, bit I think that's too
> > restrictive.
> >
>
> Is using floats really that steep a requirement?
Not really, IMHO, but I might change my mind during the next few
years, or even months. Some people have been complaining for quite a
while alreay...
> Super Collider/Max MSP
> does surprisingly well on less than the latest and greatest.
Yes, but users tend to get lazy and expect the machines to take care
of the technical details. Signal processing theory is for people of
the kind found on this list. Artists without a burning interest in
DSP programming want to do real work, and not mess with workarounds to
save bits.
> Also, hosts
> always have the option of running the plugins in less than real time.
Not always. Mixers have to run in real time, as do samplers and
synths. We're dealing with musical environments and instruments, not
only editing/rendering stations.
> > > You have it backwards - I want to make it easier to write plugins not to
> > > write hosts. It is a choice between making every plugin potentially do
> > > lots of conversions and handle lots of data types (and there will
> > > hopefully be thousands of plugins) or making a few (or at least a lot less
> > > than plugins) hosts support data conversions.
> >
> > * A plugin will not be forced to do anything but declaring what
> > format is used for each one of it's ports.
> >
> > * A plugin may be compiled in different versions with different
> > formats, if the developer is nice enough to follow the
> > recommendation of providing plugins for the major data types.
>
> My whole goal is to make it so that providing for the major data types
> only requires a recompile - as soon as the pesky integers come into the
> picture everything gets messy.
This isn't always possible, as optimized code may have to be
rewritten to take advantage of, or even work correctly with,
different formats. This is pretty obvious for integer/fixpoint
formats, but similar problems exist for FP as well.
> > * Host do not have to support *any* data types. They can get
> > away with uning only the element size field of the data type
> > descriptors, and use interface/driver plugins to communicate
> > with the outside world.
> >
> > * Hosts are required not to allow direct connection of ports
> > of different types.
> >
> > * Hosts may use an array of plugins, or inline code to convert
> > between formats as needed. This can also be done manually by
> > the user for the simplest hosts.
> >
>
> This is what scares me - it is likely that these conversions will
> have an adverse effect on sound quality and performance.
Yes, but you *can* run the plugins. People use external fx boxes with
analog interfaces or all analog cirquitry just to get the right
sound! In similar situations in virtual systems, not being able to
run a certain plugin at all is just not acceptable.
> Maybe if the
> entry and exit points of the plugin network had provisions for conversions
> . . . but this gets ugly and is, in my mind, the job of the host.
Why? IMO, the host is a scheduler and communication kernel for
plugins, not a processing engine with plugin extension capabilities.
> EEspecially since the types of conversions are likely to change rapidly
> (with all of the compressed formats, multichannel digital, etc.). This
> type of code is better placed in a library for host writers to use.
That forces users to chose hosts that happen to support the right
data formats. Again, users are *not* developers, and shouldn't be
expected to be able to tailor applications to their needs. That's
what plugins are for.
> An end user is not likely to have a strong preference about whether or not
> the plugin network is floats or roman numerals as long as it sounds good
> and all of the plugins work (I certainly feel this way when I am a user).
That's exactly why I want multiple data types and converter plugins.
The alternative is to force all hosts to support all data types, which
seems unrealistic to me, even if there would be only 2 or 3 data
types...
> What they are likely to care about is getting sound in and out of lots of
> strange formats that have no business at all in a plugin spec for any
> level (professional->fisher price tape deck). I see little hope of
> providing for these types of conversions in a clean, consistent manner
> within a, hopefully, fixed API specification.
That kind of formats belong elsewhere. The plugin API should deal
with formats that make some kind of sense to pass between *plugins*,
not the host and the outside world. (Interface/driver plugins could do
the job, but that has little to do with the plugin API. Those plugins
would use other APIs in addition to the plugin API.)
> > That is, plugins are basically coded the same way as if the data
> > type were fixed at compile time, but instead of putting the data
> > type ID in the version code, there's one field for each port. Data
> > typing is still static after compile time, but you may have ports of
> > different types on the same plugin.
> >
> > The host gets to do the work, actually. It has to compare the data
> > type IDs before connecting ports, and take some action if they're
> > different. It also has to understand the data types it wants to
> > operate on directly, unless it leaves that to special purpose
> > plugins. At can get away with understanding only one type, and using
> > converter plugins, just like when connecting incompatible ports.
> >
>
> Your whole plan is useful only if individual plugins can have multiple
> data types and this is where I am just not convinced. So far the
> rationalization for this is:
> - Preferences of coder/user
64 bits or 32 bits? You'll get lots of answers, and some will even
say "neither"...
> - Ability to support non-audio data
I'd prefer the term "non-signal data". There are quite a few
non-audio uses for DSP, but most of them still deal with the exact
same semantics - only different sample rates and sample formats.
> - Need for integer types of MMX/Altivec(?)
Fortunately, both CPUs and graphics processors (like the NVidia
GeForce 256) are getting FP SIMD or other forms of array processing,
but that's of no help to Celeron or Pentium based embedded systems.
> I think that everyone would prefer floats unless using ints would give a
> huge performance increase (but if I am wrong please tell me). I doubt
> using ints would give that big a boost except if SIMD instructions were
> used - if this is really worth it the plugin can do the conversion
> internally.
If you do that on Intel, you may as well ignore MMX altogether,
unless the processing inside the plugin is *really* heavy. Also, MMX
has too low resolution to be useful for most high end plugins (only
16 bit multiply), so it's only interesting for games sound fx engines
and the like, where all processing is still done in integer formats
anyway.
> Supporting non-audio data is most likely just going to annoy
> audio plugin writers and produce poor API for video or whatever.
Yes, I'm beginning to agree here, at least for the LADSPA API. It
doesn't provide the right kind of infrastructure to really take
advantage of this.
> If
> people are serious about making this a generic plugin api I think we
> should look into passing arbitrary, self typed messages (like Max Atoms
> but with more types). This would allow for the passing of, for example,
> pointers to buffers of video data. If the goal is going to be changed to
> making a generic API it should be made *completely* generic.
This sounds more like MuCoS, and everything does make more sense
there as well... Without the same data types for MuCoS and LADSPA,
there isn't much point in using MuCoS as an event system extension
for LADSPA. It does add sample accurate processing, but I'm not sure
the extra complexity is acceptable if that's about all you get.
Perhaps it's a better idea to simply switch to some pure MuCoS API
if you need more than what LADSPA provides?
Anyway, I'm not going to limit myself to implementing MuCoS as an
extension of LADSPA in it's current form (prov. rel 2), as it seems
to become too messy. I'll just keep working on my modofied version
with the event extension, and see if there actually *is* much point
in using the same low level stuff for both an API without events and
one with events. (The way plugins work during a single run() call is
fundamentally different, but that doesn't necessarily mean that the
interfaces have to be completely incompatible.)
//David
.- M u C o S --------------------------------. .- David Olofson ------.
| A Free/Open Multimedia | | Audio Hacker |
| Plugin and Integration Standard | | Linux Advocate |
`------------> http://www.linuxdj.com/mucos -' | Open Source Advocate |
.- A u d i a l i t y ------------------------. | Singer |
| Rock Solid Low Latency Signal Processing | | Songwriter |
`---> http://www.angelfire.com/or/audiality -' `-> david_AT_linuxdj.com -'
This archive was generated by hypermail 2b28 : Thu Mar 30 2000 - 13:27:33 EEST