Subject: Re: [linux-audio-dev] pbd's votes on LADSPA
From: Paul Barton-Davis (pbd_AT_Op.Net)
Date: Mon Mar 27 2000 - 05:14:41 EEST
>> Do you have some magic that can magically convert a sample value in the
>> range(s):
>>
>> -(2^24) ... -((2^8)+1) ... GAP ... (2^8)+1 ... (2^24) - 1
>>
>> to a scaled value without incredible processing penalties ?
>>
>> --p
>>
>
>Huh? You argue for computation in the card's native format and then you
>present a format that can't even be used mathematically without SOME kind of
>conversion ??
No, its not "the card's native format". Its just a 32 bit sample value
that just so happens to never be transmitted from the h/w containing
values between -256 .. 256. You can treat it like a 32 bit value for
almost all purposes, and indeed, everything that I know of that uses
such hardware does so (and since this is essentially just one of the
Xilinx FPGA's, quite a lot of h/w does this). For example, dB
computations just need to know that the reference level is 257 instead
of zero, etc. However, the one thing you can't do is to scale it in a
computationally trivial way to a -1,+1 range.
But anyway, my real point was that if you force a signed 24 bit
integer sample stream into a 24 bit mantissa, you can't convert it in
both directions and keep all the bits all the time. the range is
-(2^24) to +((2^24-1) because of the sign bit, and you will inevitably
end up with values that cannot be exactly represented. Consider the
very simple case: a sample value of 16777214, the maximum
theoretically possible minus one. The scaled representation of this
number (16777214/167777215) is irrational, and thus is not accurately
representable by any number of bits.
I think that what you are suggesting is something along the lines of:
normal 24 bit float representation 16777214 * (2 ^ 0)
scaled value 16777214 * (2 ^ -24)
but this is not the correct answer.
--p
This archive was generated by hypermail 2b28 : Mon Mar 27 2000 - 05:42:49 EEST