Re: [LAD] GIl, python, threading, and IPC

On Thu, Jun 18, 2009 at 11:16 AM, Patrick
Stinson<patrickstinson.lists@email-addr-hidden> wrote:
> I wrote a scripting engine for a pro audio plugin by embedding a
> CPython interpreter. Since audio sequencing hosts use separate threads
> for each audio track, loading more than one instance of my plugin
> while running scripting code causes contention on Python's GIl and
> results in CPU spikes at low latencies.
>
> While CPython is more than capable of running fast enough in the audio
> thread for control-rate (MIDI) work, the GIL is killing us. Using
> process migration to move calls to CPython to daemon processes would
> take less code than forking python itself, but the scripting engine
> includes a python extension module that exposes pointers to the C++
> classes in the audio engine. This complicates things quite a bit. The
> calls look like this:
>
> size = engine.getAudioBufferSize()
> engine.loadPatchFile(SOME_PATH)
> instrument.loadAudioFile(SOME_PATH_2)
> instrument.setVolume(.5)
>
> ...where 'engine' and 'instrument' are C++ classes in the audio engine
> that I wrapped in a python extension.
>
> I think that the biggest problem for me is tackling the complexity of
> managing new real time daemon processes for each audio track, finding
> an IPC method, and also implementing a middle-ware layer that would
> allow those synchronous calls to the engine be made back to the host.
> This is the sort of overhead that one can expect when trying to use
> process migration to work around the GIL. I consider myself an
> experienced coder, but just thinking about finding a clean, rock-solid
> daemon management and IPC mechanism that works on Mac and Windows
> freaks me out.
>
> Does anyone have any comments on this topic?
>
> cheers,
> -P
> _______________________________________________
> Linux-audio-dev mailing list
> Linux-audio-dev@email-addr-hidden
> http://lists.linuxaudio.org/mailman/listinfo/linux-audio-dev
>

My previous experience of getting around the GIL (not for audio stuff,
networking related) the multiprocessing module in python 2.6, or
pyprocessing before that is great. I had very good performance
improvements and the effort wasn't that extensive. The IPC /
management mechanisms included where enough for my needs. They are
really easy to use. For my problem performance basically scaled
linearly by number of cpus using the module. On 2-8 core machines
running many mostly io bound processes. Have you looked at stackless
as well? pyprocessing is really easy though if it works. Feel free to
give me a yell with any issues.

Loki
I am having an awesome day ;)
_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/mailman/listinfo/linux-audio-dev
Received on Thu Jun 18 08:15:02 2009