Re: [LAD] GIl, python, threading, and IPC

Thanks for the feedback, but my problem cannot be solved by creative
python code, because in order to import and use those modules my real
time C++ audio threads would have to still use the same python GIL.

On Wed, Jun 17, 2009 at 6:08 PM, Loki Davison<loki.davison@email-addr-hidden> wrote:
> On Thu, Jun 18, 2009 at 11:16 AM, Patrick
> Stinson<patrickstinson.lists@email-addr-hidden> wrote:
>> I wrote a scripting engine for a pro audio plugin by embedding a
>> CPython interpreter. Since audio sequencing hosts use separate threads
>> for each audio track, loading more than one instance of my plugin
>> while running scripting code causes contention on Python's GIl and
>> results in CPU spikes at low latencies.
>>
>> While CPython is more than capable of running fast enough in the audio
>> thread for control-rate (MIDI) work, the GIL is killing us. Using
>> process migration to move calls to CPython to daemon processes would
>> take less code than forking python itself, but the scripting engine
>> includes a python extension module that exposes pointers to the C++
>> classes in the audio engine. This complicates things quite a bit. The
>> calls look like this:
>>
>> size = engine.getAudioBufferSize()
>> engine.loadPatchFile(SOME_PATH)
>> instrument.loadAudioFile(SOME_PATH_2)
>> instrument.setVolume(.5)
>>
>> ...where 'engine' and 'instrument' are C++ classes in the audio engine
>> that I wrapped in a python extension.
>>
>> I think that the biggest problem for me is tackling the complexity of
>> managing new real time daemon processes for each audio track, finding
>> an IPC method, and also implementing a middle-ware layer that would
>> allow those synchronous calls to the engine be made back to the host.
>> This is the sort of overhead that one can expect when trying to use
>> process migration to work around the GIL. I consider myself an
>> experienced coder, but just thinking about finding a clean, rock-solid
>> daemon management and IPC mechanism that works on Mac and Windows
>> freaks me out.
>>
>> Does anyone have any comments on this topic?
>>
>> cheers,
>> -P
>> _______________________________________________
>> Linux-audio-dev mailing list
>> Linux-audio-dev@email-addr-hidden
>> http://lists.linuxaudio.org/mailman/listinfo/linux-audio-dev
>>
>
> My previous experience of getting around the GIL (not for audio stuff,
> networking related) the multiprocessing module in python 2.6, or
> pyprocessing before that is great. I had very good performance
> improvements and the effort wasn't that extensive. The IPC /
> management mechanisms  included  where enough for my needs. They are
> really easy to use. For my problem performance basically scaled
> linearly by number of cpus using the module. On 2-8 core machines
> running many mostly io bound processes. Have you looked at stackless
> as well? pyprocessing is really easy though if it works. Feel free to
> give me a yell with any issues.
>
> Loki
> I am having an awesome day ;)
>
_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev@email-addr-hidden
http://lists.linuxaudio.org/mailman/listinfo/linux-audio-dev
Received on Thu Jun 18 08:15:02 2009