Re: [LAD] "bleeding edge html5" has interesting Audio APIs

From: Nick Copeland <nickycopeland@email-addr-hidden>
Date: Tue Nov 22 2011 - 23:01:55 EET

This is sent with apologies to Fons who now has about 50,000 copies of the
same submit grace of hotmail (great web interface).

> From: fons@email-addr-hidden
> To: nickycopeland@email-addr-hidden
> Subject: Re: [LAD] "bleeding edge html5" has interesting Audio APIs
>
> On Tue, Nov 22, 2011 at 06:05:51PM +0100, Nick Copeland wrote:
>
> > Which toolkit is this? Having the graphical position of the slider/pot define its
> > value sounds a little broken.
>
> ?? I must be missing something essential...
>
> For example, you have a slider that is 100 pixels long, it has
> 101 possible positions, each one of those corresponds to some
> parameter value which is recomputed whenever you move the slider.
>
> Other approaches are possible, but this is AFAIK what most
> toolkit sliders do. So the graphical state determines the
> parameter value...
> Igor sanin wrote:
> I may be missing something here. The purpose of a slider is to set a
> value. The position of the slider is a graphical representation of
> that value.
So let's talk about the tail wagging the dog.

An application needs input and output, toolkits are used for all of this for lots
of reasons such portability, segregation of duties, whatever. The toolkit needs to
provide input to the app, the app needs to generate the output.

Input to an app are the cursor motion, keystrokes, on this list there is also MIDI
and similar control channels.

Is audio the only output? I kind of disagree that the output is only audio since
we are looking at whatever we are working with as much as we are listening to it.
The graphical output is as important as the audio output, you might disagree but
at least you can say that minimally these are both outputs rather than inputs - they
are what you see and what you hear from the application.

Input to the application should be values that are extracted from the pointer
positions in relation to what they are controlling. They are inputs, they have nothing
to do with slider positions on the screen just as they have nothing to do with the
actual generation of the sound, they are floating point numbers from the user interface
into the program.

The program/application uses these numbers to change audible output since some
control value has changed, they should also define the graphical output since, let's
be honest: the graphical display is output, not input.

If you are using a toolkit that has a data flow of the following:

      pointer motion->graphical display->values->application->output

Well, basically that is broken as you have a flow that is

      input->output->input->application->output

invariably that is going to lead to issues. The tail (the toolkit) is wagging the dog
(the application) as it imposes restrictions on the values the application is allowed
to see.

In my opinion (ok, it is only opinion) is that the correct flow is

     input->application->output

It is irrelevant whether the output we are discussing is graphical or audible. The
API/library/toolkit should give values to the application, the application should
decide how these values need to be processed in the audio DSP and also how the
toolkit should represent them to the visible user interface. Remember, the toolkit
has no inclination of what it is being used for so why should it define what values it
passes to its owner?

> Then when you numerically enter a value, the slider will
> move to the nearest position. But in most cases that won't
> correspond to the value you entered, but to something near.
> So as soon as you move the slider, you're back to one of
> its 101 possible values, and you can't return to the typed-in
> one except by typing it again.

With this model, when you enter a value that value will be given to your application
and it can use it as it sees fit to generate sound: it uses the value to tell the DSP
code what it has to generate. It should also use similar processes to tell the other
output channel, the graphical one, what is should display to represent that value
in the graphical interface. The graphical output should not define the input values
to the application.

Just as the audio is output, so are the graphics. Your slider might go from 0 to 101,
or if the GUI is resized them from 0 to 201. The input channel has the responsibility
to normalise that back to 0.0 to 1.0 and give it to the application. There is no reason
that the toolkit be based on pixels, if it is then it is broken as it does not have any
abstraction that makes it generally useful.

> > A better solution would be for the application callback to be given the actual
> > value and it decide what that means for whatever it is controlling and how the
> > graphical output should be depicted. The tail should not be wagging the dog
> > but again that might depend on the toolkit(?)
>
> Are you referring to the case where the application needs less resolution
> than the widget provides ? I that case it can compute its nearest preferred
> value and make the widget display that (as soon as the user releases the
> widget).

I was referring to the application always being given a float of the value of the
control - 0.0 to 1.0 representing zero to full throw. The application decides how
this should be used, if it is continuous so be it. If it is a stepped control (semitone
tuning) then the application should send the stepped value to the audio engine
and request the graphical engine depict a 'step' in its display - the graphical
output now has steps just as the audio output does. If there is any quantisation
that has to be done then let the application control that, not the toolkit. The app
is not responsible for what gets drawn but it does give directives about what it
wants to have displayed that reflect the parameter being represented.

Back in the '90s I was writing a mixing desk emulator call S-Lab which had direct
to disk sampling and some basic (cheesy) editing. It used a graphics toolkit.
Somewhere around 2001/2002 I started working on what become bristol, a set of
synths for this Lab but the limitations of the toolkit became an issue, the code I
wanted had to bend over backwards to get the desired output - the graphics were
grisly.

So I dumped the toolkit and wrote my own that was based on a input-app-output
paradigm that left the application in control of what is wants to do. This is how
brighton works, it's a holistic approach, the user interface becomes as much what
they see as what they hear - the different output channels.

I really do not see why a toolkit should define how it is used and hearing people talk
about pixel boundary limitations makes me cringe.

> Igor sanin wrote
> On a related note, the probability that a potentiometer can be set to exactly the same
> resistance twice is close to zero. This does not make potentiometers less prevalent in
> musical applications.
Yeah, I like that. All the small discrepancies are a large part of why analogue was/is so nice.

> > On a related note, Fons, weren't you actually working recently on some subpixel
> > controllers? How were they implemented?
>
> Yes, I did a mixer gain fader that has subpixel resolution, not only the value
> but also the graphical state itself. Nothing special. If you create the 'knob'
> pictures using e.g. cairo (pycairo in this case), it's easy to make it move by
> a fraction of a pixel.
>
> But I also found that for a 250 pixel long gain fader it's just overkill.
> The one I use has exact 0.25 dB steps over the upper part of its range. Below
> that the steps size increases, and you get values that are not k * 0.25 dB,
> but below -30 dB or so that doesn't matter much.

What about 25 pixel long faders? Don't fanning controls give you back the resolution
you need here? I think I must be missing something in the pixel level controls with
respect to how the toolkits work, the issue is that I have not used anybody else'
graphical toolkit for over 10 years now.

"we have to make sure the old choice [Windows] doesn't disappear”.
Jim Wong, president of IT products, Acer

From: nickycopeland@email-addr-hidden
To: fons@email-addr-hidden; linux-audio-dev@email-addr-hidden
Date: Tue, 22 Nov 2011 18:05:51 +0100
Subject: Re: [LAD] "bleeding edge html5" has interesting Audio APIs

> From: fons@email-addr-hidden
> To: linux-audio-dev@email-addr-hidden
> Subject: Re: [LAD] "bleeding edge html5" has interesting Audio APIs
>
> On Tue, Nov 22, 2011 at 05:59:40PM +0400, Alexandre Prokoudine wrote:
> > On Tue, Nov 22, 2011 at 5:54 PM, Fons Adriaensen wrote:
> >
> > >> For darktable we examined the slider from phat and created a similar
> > >> new, more usable widget which combines a label and a slider. You can
> > >> enter precise value after a right click inside the slider area, and
> > >> you can use pretty much anything as displayed unit: %, ", dB, px...
> > >> whatever. Here is an example:
> > >
> > > Not a real solution. You now have a value that is not represented
> > > by a slider position.
> >
> > Er... WHAT???
> >
> > It absolutely IS represented.
> >
> > 1. Right click
> > 2. Enter new value
> > 3. Press Enter to confirm
> > 4. Slider updates
>
> That is assuming that the slider has infinite resolution.

Which toolkit is this? Having the graphical position of the slider/pot define its
value sounds a little broken.

> In 4. the slider will move to the nearest (pixel) position.
> You could paint it with higher resolution, but mouse gestures
> are still quantised per pixel. If you move the slider 1 pixel

Touchpad interfaces support subpixel (floating point) coordinates based on an
interpolation of where somebodies fat greasy digit smudges the screen, it is
actually quite useful. HTML5 also transports pointer motion as floats for this
reason. Am an NOT advocating its use, just stating that subpixel is there.

> up and down again you don't get the value that was typed in,
> unless it happens to be one corressponding to a pixel.

Again, why does the graphical output have to define the value of the input?
Surely that is a limitation of the toolkit?

> The only solution is to ensure that slider positions correspond
> to values that make sense for the parameter being controlled,
> e.g. exact semitones (or fractions thereof) for a VCO frequency,
> 1/5 dB steps for the top half of a fader, etc. And if they do
> then you don't need the text input.

A better solution would be for the application callback to be given the actual
value and it decide what that means for whatever it is controlling and how the
graphical output should be depicted. The tail should not be wagging the dog
but again that might depend on the toolkit(?)

On a related note, Fons, weren't you actually working recently on some subpixel
controllers? How were they implemented?

Kind regards, nick.
                                               

_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/listinfo/linux-audio-dev

_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/listinfo/linux-audio-dev
Received on Wed Nov 23 00:15:04 2011

This archive was generated by hypermail 2.1.8 : Wed Nov 23 2011 - 00:15:04 EET